Nov 02 22:02:29.184774 ip-10-0-140-221 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 22:02:29.184786 ip-10-0-140-221 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 22:02:29.184793 ip-10-0-140-221 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 22:02:29.185070 ip-10-0-140-221 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 22:02:39.304684 ip-10-0-140-221 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 22:02:39.304706 ip-10-0-140-221 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 67df9700f71d470ea2567bdade58927e -- Nov 02 22:05:00.589884 ip-10-0-140-221 systemd[1]: Starting Kubernetes Kubelet... Nov 02 22:05:01.096930 ip-10-0-140-221 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:05:01.096930 ip-10-0-140-221 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 22:05:01.096930 ip-10-0-140-221 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:05:01.096930 ip-10-0-140-221 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:05:01.100589 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100434 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:05:01.100589 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100589 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:05:01.100589 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100594 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100597 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100600 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100603 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100606 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100608 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100611 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100614 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100618 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100621 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100625 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100628 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100630 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100633 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100636 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100638 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100641 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100644 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100647 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100650 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:05:01.100692 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100652 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100655 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100659 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100662 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100665 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100672 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100675 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100678 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100681 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100684 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100686 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100689 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100692 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100695 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100698 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100701 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100704 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100707 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100710 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:05:01.101157 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100712 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100716 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100718 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100722 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100725 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100728 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100731 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100734 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100737 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100740 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100742 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100745 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100748 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100751 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100753 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100756 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100759 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100763 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100766 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100769 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:05:01.101674 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100773 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100776 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100779 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100782 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100785 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100788 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100790 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100793 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100798 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100802 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100806 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100809 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100812 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100815 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100818 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100822 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100827 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100831 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100834 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:05:01.102158 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100837 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100840 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100843 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100846 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100848 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100851 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100854 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100857 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100859 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100862 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100865 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100868 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100871 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100874 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100876 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100880 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100883 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100886 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100889 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:05:01.102635 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100892 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100895 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100897 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100900 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100903 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100906 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.100909 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101767 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101777 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101780 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101784 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101787 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101790 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101794 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101797 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101799 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101802 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101805 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101808 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101811 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:05:01.103100 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101814 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101817 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101820 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101823 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101826 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101829 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101832 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101834 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101837 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101841 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101847 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101851 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101854 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101857 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101860 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101863 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101866 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101868 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101871 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:05:01.103599 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101874 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101876 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101879 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101882 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101884 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101887 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101890 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101893 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101896 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101898 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101901 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101904 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101906 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101912 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101915 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101918 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101921 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101923 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101926 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101929 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:05:01.104065 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101931 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101934 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101937 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101939 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101942 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101945 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101948 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101950 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101953 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101956 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101959 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101961 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101964 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101966 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101969 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101971 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101974 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101977 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101979 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:05:01.104562 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101982 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101985 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101988 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101990 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101993 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.101996 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102001 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102004 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102006 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102011 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102013 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102016 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102019 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102021 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102024 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102027 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102029 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102032 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102035 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:05:01.105024 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102037 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102040 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102042 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102045 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102048 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102050 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102053 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102056 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102058 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102061 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102063 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102066 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102071 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102075 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102078 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102081 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102625 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102631 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102635 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:05:01.105661 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102640 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102643 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102647 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102651 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102654 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102657 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102660 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102663 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102666 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102668 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102671 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102674 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102677 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102680 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102683 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102686 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102688 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102691 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102694 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102696 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:05:01.106114 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102699 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102702 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102705 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102707 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102710 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102712 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102715 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102720 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102722 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102725 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102727 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102730 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102733 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102736 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102739 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102742 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102744 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102747 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102751 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102754 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:05:01.106615 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102756 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102759 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102761 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102764 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102767 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102769 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102772 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102774 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102777 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102780 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102782 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102785 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102788 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102791 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102793 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102796 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102798 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102809 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102812 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:05:01.107154 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102815 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102825 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102828 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102831 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102833 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102840 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102844 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102847 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102851 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102854 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102859 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102863 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102866 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102869 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102871 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102874 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102877 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102880 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102882 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:05:01.107621 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102885 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102888 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102891 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102893 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102896 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102898 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102901 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102903 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102906 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102909 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102912 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102914 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102917 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102920 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102922 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102932 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102935 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102938 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102940 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102943 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:05:01.108216 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102945 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102948 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102950 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102953 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.102961 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104036 2581 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104058 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104067 2581 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104073 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104079 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104083 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104089 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104094 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104098 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104101 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104106 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104110 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104113 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104117 2581 flags.go:64] FLAG: --cgroup-root="" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104120 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104123 2581 flags.go:64] FLAG: --client-ca-file="" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104127 2581 flags.go:64] FLAG: --cloud-provider="external" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104130 2581 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104142 2581 flags.go:64] FLAG: --cluster-domain="" Nov 02 22:05:01.108704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104145 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104149 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104152 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104156 2581 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104169 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104173 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104176 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104180 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104183 2581 flags.go:64] FLAG: --contention-profiling="false" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104187 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104190 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104194 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104197 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104202 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104206 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104209 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104213 2581 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104216 2581 flags.go:64] FLAG: --enable-server="true" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104220 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104227 2581 flags.go:64] FLAG: --event-burst="100" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104231 2581 flags.go:64] FLAG: --event-qps="50" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104234 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104237 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104241 2581 flags.go:64] FLAG: --eviction-hard="" Nov 02 22:05:01.109277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104246 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104250 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104253 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104256 2581 flags.go:64] FLAG: --eviction-soft="" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104260 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104263 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104266 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104269 2581 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104272 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104275 2581 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104278 2581 flags.go:64] FLAG: --feature-gates="" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104283 2581 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104286 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104296 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104299 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104303 2581 flags.go:64] FLAG: --healthz-port="10248" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104306 2581 flags.go:64] FLAG: --help="false" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104309 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104314 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104317 2581 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104320 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104324 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104332 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 22:05:01.109883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104337 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104340 2581 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104343 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104347 2581 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104350 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104353 2581 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104356 2581 flags.go:64] FLAG: --kube-reserved="" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104359 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104363 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104366 2581 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104369 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104372 2581 flags.go:64] FLAG: --lock-file="" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104376 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104379 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104382 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104389 2581 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104392 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104396 2581 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104399 2581 flags.go:64] FLAG: --logging-format="text" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104402 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104406 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104409 2581 flags.go:64] FLAG: --manifest-url="" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104412 2581 flags.go:64] FLAG: --manifest-url-header="" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104421 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104424 2581 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 22:05:01.110438 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104429 2581 flags.go:64] FLAG: --max-pods="110" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104432 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104435 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104439 2581 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104442 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104445 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104450 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104454 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104463 2581 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104466 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104470 2581 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104473 2581 flags.go:64] FLAG: --pod-cidr="" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104477 2581 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104480 2581 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104484 2581 flags.go:64] FLAG: --pods-per-core="0" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104487 2581 flags.go:64] FLAG: --port="10250" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104490 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104494 2581 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-03af722d6dfd167a6" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104497 2581 flags.go:64] FLAG: --qos-reserved="" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104500 2581 flags.go:64] FLAG: --read-only-port="10255" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104503 2581 flags.go:64] FLAG: --register-node="true" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104506 2581 flags.go:64] FLAG: --register-with-taints="" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104511 2581 flags.go:64] FLAG: --registry-burst="10" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104514 2581 flags.go:64] FLAG: --registry-qps="5" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104527 2581 flags.go:64] FLAG: --reserved-cpus="" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104531 2581 flags.go:64] FLAG: --reserved-memory="" Nov 02 22:05:01.111063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104535 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104539 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104542 2581 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104545 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104548 2581 flags.go:64] FLAG: --runonce="false" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104552 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104556 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104559 2581 flags.go:64] FLAG: --seccomp-default="false" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104562 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104566 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104569 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104573 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104580 2581 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104584 2581 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104587 2581 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104590 2581 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104594 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104597 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104600 2581 flags.go:64] FLAG: --system-cgroups="" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104603 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104610 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104613 2581 flags.go:64] FLAG: --tls-cert-file="" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104617 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104622 2581 flags.go:64] FLAG: --tls-min-version="" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104625 2581 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 22:05:01.111912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104628 2581 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104632 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104635 2581 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104638 2581 flags.go:64] FLAG: --v="2" Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104644 2581 flags.go:64] FLAG: --version="false" Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104648 2581 flags.go:64] FLAG: --vmodule="" Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104653 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.104657 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104808 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104813 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104816 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104818 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104821 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104825 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104827 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104830 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104833 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104836 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104839 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104844 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104847 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104850 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:05:01.113004 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104853 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104856 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104859 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104861 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104864 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104867 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104870 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104874 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104878 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104881 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104884 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104887 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104890 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104892 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104895 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104898 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104901 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104903 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104906 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:05:01.113963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104909 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104912 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104915 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104917 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104920 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104924 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104928 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104932 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104935 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104938 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104943 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104946 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104949 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104952 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104954 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104957 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104960 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104963 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104965 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:05:01.114845 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104968 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104970 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104973 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104976 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104979 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104981 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104984 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104987 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104989 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104992 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104995 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.104997 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105000 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105002 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105005 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105008 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105011 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105013 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105016 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105020 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:05:01.115693 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105022 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105025 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105028 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105033 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105036 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105039 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105042 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105045 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105047 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105050 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105053 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105055 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105058 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105061 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105063 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105066 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105069 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105072 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105075 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:05:01.116597 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105077 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:05:01.117280 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105080 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:05:01.117280 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105083 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:05:01.117280 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105085 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:05:01.117280 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105088 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:05:01.117280 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105091 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:05:01.117280 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105093 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:05:01.117280 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105096 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:05:01.117280 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105099 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:05:01.117280 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105101 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:05:01.117280 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105104 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:05:01.117280 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105107 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:05:01.117280 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105109 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:05:01.117280 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105112 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:05:01.117280 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105115 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:05:01.117280 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.105144 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:05:01.117280 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105185 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105191 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105195 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105198 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105201 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105203 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105206 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105209 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105211 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105214 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105217 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105220 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105223 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105225 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105228 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105231 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105234 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105236 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105239 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105242 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:05:01.117714 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105244 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105247 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105250 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105252 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105255 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105258 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105260 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105264 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105266 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105269 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105273 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105277 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105282 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105286 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105289 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105292 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105295 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105298 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105300 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:05:01.118483 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105303 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105307 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105309 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105312 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105315 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105318 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105321 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105323 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105327 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105331 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105334 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105337 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105342 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105345 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105348 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105350 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105353 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105356 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105359 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:05:01.119207 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105362 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105364 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105367 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105370 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105373 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105376 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105380 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105384 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105387 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105396 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105399 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105402 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105405 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105407 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105410 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105413 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105416 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105418 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105421 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105424 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:05:01.119680 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105426 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105429 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105432 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105434 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105437 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105440 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105443 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105446 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105448 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105451 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105454 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105457 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105459 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105462 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105464 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105467 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105470 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105472 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105478 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:05:01.120198 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105482 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105485 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105488 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105491 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105493 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105496 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105499 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105501 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.105504 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.114597 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.114628 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.114646 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.114654 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114753 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114763 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114768 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114773 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114779 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114784 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:05:01.120669 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114789 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114794 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114799 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114812 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114817 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114822 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114826 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114830 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114836 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114840 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114844 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114848 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114853 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114856 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114860 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114864 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114868 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114872 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114876 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114881 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:05:01.121136 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114885 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114890 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114894 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114908 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114913 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114918 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114923 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114928 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114932 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114936 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114941 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114947 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114955 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114960 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114965 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114969 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114973 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114978 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114982 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:05:01.121639 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114987 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114991 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114995 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.114999 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115003 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115008 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115012 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115017 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115021 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115025 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115030 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115034 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115038 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115043 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115047 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115051 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115057 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115070 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115075 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115079 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:05:01.122083 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115084 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115088 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115092 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115096 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115101 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115105 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115110 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115115 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115119 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115124 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115128 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115134 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115140 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115145 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115150 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115154 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115159 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115163 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115168 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:05:01.122572 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115172 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115177 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115181 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115185 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115189 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115194 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115199 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115203 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115207 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115213 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115217 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115232 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115236 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115241 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115245 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115249 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115254 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115258 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115262 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115266 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:05:01.123029 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115271 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:05:01.123512 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115275 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:05:01.123512 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.115323 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:05:01.123512 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115640 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:05:01.123512 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115649 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:05:01.123512 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115655 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:05:01.123512 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115661 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:05:01.123512 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115666 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:05:01.123512 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115670 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:05:01.123512 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115677 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:05:01.123512 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115683 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:05:01.123512 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115688 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:05:01.123512 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115692 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:05:01.123512 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115696 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:05:01.123512 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115700 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:05:01.123512 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115704 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:05:01.123512 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115709 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115713 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115717 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115721 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115726 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115731 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115735 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115740 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115753 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115758 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115763 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115767 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115772 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115776 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115781 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115785 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115789 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115794 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115798 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115802 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:05:01.124030 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115807 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115811 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115815 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115820 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115824 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115828 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115832 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115837 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115841 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115845 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115850 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115854 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115858 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115863 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115867 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115871 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115876 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115880 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115885 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:05:01.124493 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115892 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115898 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115912 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115917 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115922 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115926 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115930 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115934 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115939 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115943 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115947 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115951 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115955 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115960 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115964 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115969 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115973 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115977 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115981 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115986 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:05:01.124963 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115991 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.115995 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116000 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116004 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116009 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116013 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116017 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116022 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116026 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116030 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116034 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116040 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116045 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116049 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116053 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116065 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116070 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116074 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116078 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116083 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:05:01.125434 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116087 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:05:01.125915 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116092 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:05:01.125915 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116096 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:05:01.125915 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116100 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:05:01.125915 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116105 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:05:01.125915 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116109 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:05:01.125915 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116113 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:05:01.125915 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116117 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:05:01.125915 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116121 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:05:01.125915 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116125 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:05:01.125915 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116130 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:05:01.125915 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116135 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:05:01.125915 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116140 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:05:01.125915 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116144 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:05:01.125915 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.116184 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:05:01.125915 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116264 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:05:01.125915 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116271 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116276 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116280 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116284 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116289 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116293 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116297 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116302 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116307 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116312 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116316 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116320 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116333 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116338 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116342 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116347 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116351 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116357 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116362 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:05:01.126286 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116368 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116373 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116377 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116382 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116386 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116390 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116395 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116399 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116403 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116408 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116412 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116420 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116424 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116429 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116433 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116437 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116442 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116446 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116451 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:05:01.126758 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116455 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116460 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116465 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116469 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116473 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116478 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116482 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116495 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116500 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116504 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116509 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116513 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116533 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116538 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116543 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116548 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116552 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116556 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116561 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:05:01.127236 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116566 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116570 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116574 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116579 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116583 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116589 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116595 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116602 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116607 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116611 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116616 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116620 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116624 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116628 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116633 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116637 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116642 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116646 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116650 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:05:01.127754 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116655 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116659 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116672 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116677 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116682 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116686 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116691 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116697 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116701 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116705 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116709 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116713 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116718 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116726 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116730 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116734 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116739 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116743 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116749 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116753 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:05:01.128204 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116758 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:05:01.128746 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116762 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:05:01.128746 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116766 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:05:01.128746 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116771 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:05:01.128746 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116775 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:05:01.128746 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116779 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:05:01.128746 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116784 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:05:01.128746 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116788 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:05:01.128746 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:01.116792 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:05:01.128746 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.116900 2581 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 22:05:01.128746 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.121853 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 22:05:01.128746 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.122979 2581 server.go:1014] "Starting client certificate rotation" Nov 02 22:05:01.128746 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.123075 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:05:01.128746 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.123122 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:05:01.128746 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.127411 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:05:01.132972 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.132949 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:05:01.143582 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.143555 2581 log.go:25] "Validated CRI v1 runtime API" Nov 02 22:05:01.147876 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.147857 2581 log.go:25] "Validated CRI v1 image API" Nov 02 22:05:01.148688 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.148656 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 22:05:01.151561 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.151507 2581 fs.go:135] Filesystem UUIDs: map[0d690c39-df18-469b-a300-46c8be12f295:/dev/nvme0n1p3 79757a5c-3a20-4ae0-9226-5f2097252f39:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 02 22:05:01.151667 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.151559 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 22:05:01.158509 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.158360 2581 manager.go:217] Machine: {Timestamp:2026-11-02 22:05:01.15667177 +0000 UTC m=+0.422891342 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095738 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23bd2316f94f86057c183258a0336f SystemUUID:ec23bd23-16f9-4f86-057c-183258a0336f BootID:67df9700-f71d-470e-a256-7bdade58927e Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:eb:79:0c:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:eb:79:0c:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:d5:b8:1d:ea:a6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 22:05:01.158509 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.158501 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 22:05:01.158636 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.158616 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 22:05:01.161866 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.161825 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 22:05:01.162021 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.161866 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-221.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 22:05:01.162799 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.162786 2581 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 22:05:01.162854 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.162808 2581 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 22:05:01.162854 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.162839 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:05:01.162854 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.162852 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:05:01.163883 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.163870 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 22:05:01.167971 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.167954 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 22:05:01.168127 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.168117 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 22:05:01.168931 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.168919 2581 kubelet.go:493] "Attempting to sync node with API server" Nov 02 22:05:01.168967 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.168935 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 22:05:01.168967 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.168952 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 22:05:01.168967 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.168963 2581 kubelet.go:405] "Adding apiserver pod source" Nov 02 22:05:01.169068 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.168975 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 22:05:01.169420 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.169407 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:05:01.173298 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.173273 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 22:05:01.179503 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.179479 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:05:01.180672 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.180652 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 22:05:01.180775 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.180686 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 22:05:01.180895 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.180883 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 22:05:01.180932 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.180901 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 22:05:01.180932 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.180907 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 22:05:01.180932 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.180913 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 22:05:01.180932 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.180919 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 22:05:01.180932 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.180926 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 22:05:01.181513 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.181504 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 22:05:01.181560 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.181514 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 22:05:01.181560 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.181538 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 22:05:01.181560 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.181544 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 22:05:01.181560 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.181553 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 22:05:01.181712 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.181563 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 22:05:01.182364 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.182354 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 22:05:01.182364 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.182364 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 22:05:01.186264 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.186246 2581 server.go:1263] "Started kubelet" Nov 02 22:05:01.186415 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.186335 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 22:05:01.186478 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.186376 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 22:05:01.186478 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.186460 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 22:05:01.187031 ip-10-0-140-221 systemd[1]: Started Kubernetes Kubelet. Nov 02 22:05:01.187859 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.187842 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 22:05:01.189514 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.189497 2581 server.go:317] "Adding debug handlers to kubelet server" Nov 02 22:05:01.192887 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:01.192863 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 22:05:01.194563 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.194547 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 22:05:01.195664 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.195648 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 22:05:01.195835 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:01.195815 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-221.us-west-2.compute.internal\" not found" Nov 02 22:05:01.196484 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.196450 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 22:05:01.196484 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.196454 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 22:05:01.196631 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.196497 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 22:05:01.196631 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.196537 2581 factory.go:55] Registering systemd factory Nov 02 22:05:01.196631 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.196554 2581 factory.go:223] Registration of the systemd container factory successfully Nov 02 22:05:01.196749 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.196649 2581 reconstruct.go:97] "Volume reconstruction finished" Nov 02 22:05:01.196749 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.196659 2581 reconciler.go:29] "Reconciler: start to sync state" Nov 02 22:05:01.196802 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.196780 2581 factory.go:153] Registering CRI-O factory Nov 02 22:05:01.196802 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.196793 2581 factory.go:223] Registration of the crio container factory successfully Nov 02 22:05:01.196858 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.196841 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 22:05:01.196887 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.196858 2581 factory.go:103] Registering Raw factory Nov 02 22:05:01.196887 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.196875 2581 manager.go:1199] Started watching for new ooms in manager Nov 02 22:05:01.197821 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.197804 2581 manager.go:319] Starting recovery of all containers Nov 02 22:05:01.210597 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.210577 2581 manager.go:324] Recovery completed Nov 02 22:05:01.214855 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.214839 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:05:01.216877 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.216855 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:05:01.216957 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.216894 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:05:01.216957 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.216906 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:05:01.217466 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.217450 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:05:01.217557 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.217451 2581 cpu_manager.go:226] "Starting" policy="none" Nov 02 22:05:01.217557 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.217506 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 22:05:01.217654 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.217568 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 22:05:01.218124 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.218109 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:05:01.218209 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.218141 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:05:01.218209 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.218156 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:05:01.218912 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.218899 2581 policy_none.go:50] "Start" Nov 02 22:05:01.218974 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.218918 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 22:05:01.218974 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.218929 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 22:05:01.220254 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.220243 2581 policy_none.go:44] "Start" Nov 02 22:05:01.265427 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.265259 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 22:05:01.270177 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.270162 2581 manager.go:341] "Starting Device Plugin manager" Nov 02 22:05:01.271014 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:01.270996 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 22:05:01.271079 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.271016 2581 server.go:87] "Starting device plugin registration server" Nov 02 22:05:01.271261 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.271250 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 22:05:01.271305 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.271270 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 22:05:01.271413 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.271387 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 22:05:01.271509 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.271498 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 22:05:01.271577 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.271510 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 22:05:01.272037 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:01.272009 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 22:05:01.272124 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:01.272061 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-221.us-west-2.compute.internal\" not found" Nov 02 22:05:01.322163 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.322134 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-221.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 22:05:01.322163 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:01.322141 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-221.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 22:05:01.323677 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:01.322126 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-221.us-west-2.compute.internal.18e459a15d0f9edd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-221.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-221.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 22:05:01.186211549 +0000 UTC m=+0.452431121,LastTimestamp:2026-11-02 22:05:01.186211549 +0000 UTC m=+0.452431121,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-221.us-west-2.compute.internal,}" Nov 02 22:05:01.344304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.344266 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 22:05:01.345562 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.345546 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 22:05:01.345623 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.345568 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 22:05:01.345623 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.345599 2581 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 22:05:01.345698 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:01.345639 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 22:05:01.371768 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.371681 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:05:01.372592 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.372574 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:05:01.372685 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.372602 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:05:01.372685 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.372612 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:05:01.372685 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:01.372642 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:01.387465 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:01.387428 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 22:05:01.387465 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:01.387438 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 22:05:01.387747 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:01.387399 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-221.us-west-2.compute.internal.18e459a15ee3960e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-221.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-221.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-221.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 22:05:01.216880142 +0000 UTC m=+0.483099712,LastTimestamp:2026-11-02 22:05:01.216880142 +0000 UTC m=+0.483099712,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-221.us-west-2.compute.internal,}" Nov 02 22:05:01.387747 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:01.387585 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-221.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 22:05:03.134809 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.134776 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 22:05:03.134987 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.134827 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4l5p8" Nov 02 22:05:03.140575 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.140546 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal"] Nov 02 22:05:03.140705 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.140614 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:05:03.142884 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.142847 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:05:03.142884 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.142890 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:05:03.143042 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.142901 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:05:03.144437 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.144422 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:05:03.144592 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.144574 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.144670 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.144608 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:05:03.145265 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.145249 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:05:03.145345 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.145276 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:05:03.145345 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.145287 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:05:03.145345 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.145248 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:05:03.145431 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.145350 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:05:03.145431 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.145361 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:05:03.147376 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.147359 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.147443 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.147387 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:05:03.148221 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.148206 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:05:03.148310 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.148239 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:05:03.148310 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.148256 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:05:03.165032 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:03.165010 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-221.us-west-2.compute.internal\" not found" node="ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.166938 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:03.166833 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-221.us-west-2.compute.internal.18e459a15ee3e1da default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-221.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-221.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-221.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 22:05:01.216899546 +0000 UTC m=+0.483119117,LastTimestamp:2026-11-02 22:05:01.216899546 +0000 UTC m=+0.483119117,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-221.us-west-2.compute.internal,}" Nov 02 22:05:03.167355 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:03.167334 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-221.us-west-2.compute.internal\" not found" node="ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.192752 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.192719 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c91a35388f9dd8d39adce6d4d82b0bac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"c91a35388f9dd8d39adce6d4d82b0bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.192752 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.192752 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c91a35388f9dd8d39adce6d4d82b0bac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"c91a35388f9dd8d39adce6d4d82b0bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.192971 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.192773 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8332e3a50281a723271a56acbbdc782f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"8332e3a50281a723271a56acbbdc782f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.245838 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.245808 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:05:03.246135 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:03.246115 2581 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-221.us-west-2.compute.internal\" not found" node="ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.293447 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.293407 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c91a35388f9dd8d39adce6d4d82b0bac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"c91a35388f9dd8d39adce6d4d82b0bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.293447 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.293447 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c91a35388f9dd8d39adce6d4d82b0bac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"c91a35388f9dd8d39adce6d4d82b0bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.293675 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.293502 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c91a35388f9dd8d39adce6d4d82b0bac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"c91a35388f9dd8d39adce6d4d82b0bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.293675 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.293559 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c91a35388f9dd8d39adce6d4d82b0bac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"c91a35388f9dd8d39adce6d4d82b0bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.293675 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.293592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8332e3a50281a723271a56acbbdc782f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"8332e3a50281a723271a56acbbdc782f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.293675 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.293620 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8332e3a50281a723271a56acbbdc782f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"8332e3a50281a723271a56acbbdc782f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.325756 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:03.325658 2581 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-221.us-west-2.compute.internal\" not found" node="ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.446424 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.446385 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:05:03.448218 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.448193 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:05:03.448327 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.448237 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:05:03.448327 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.448251 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:05:03.448327 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.448288 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.469458 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.469426 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.472167 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.472141 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.506388 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.506360 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:03.506388 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:03.506389 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-221.us-west-2.compute.internal\": node \"ip-10-0-140-221.us-west-2.compute.internal\" not found" Nov 02 22:05:03.629280 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:03.629198 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-221.us-west-2.compute.internal\" not found" Nov 02 22:05:03.730149 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:03.730115 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-221.us-west-2.compute.internal\" not found" Nov 02 22:05:03.817401 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.817364 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 22:05:03.817904 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:03.817542 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 22:05:03.817904 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:03.817587 2581 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ac0a2f50c2ab94c12807d3b1c760afba-8cfc36a42c20be03.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-140-221.us-west-2.compute.internal.18e459a15ee40a87\": read tcp 10.0.140.221:39948->3.131.201.109:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-140-221.us-west-2.compute.internal.18e459a15ee40a87 default 5313 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-221.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-221.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-221.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 22:05:01 +0000 UTC,LastTimestamp:2026-11-02 22:05:01.372616612 +0000 UTC m=+0.638836183,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-221.us-west-2.compute.internal,}" Nov 02 22:05:03.830713 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:03.830680 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-221.us-west-2.compute.internal\" not found" Nov 02 22:05:03.931675 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:03.931606 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-221.us-west-2.compute.internal\" not found" Nov 02 22:05:04.032088 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:04.032054 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-221.us-west-2.compute.internal\" not found" Nov 02 22:05:04.083276 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.083250 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:05:04.084340 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.084325 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:05:04.084421 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.084354 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:05:04.084421 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.084364 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:05:04.132600 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:04.132558 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-221.us-west-2.compute.internal\" not found" Nov 02 22:05:04.139751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.139710 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 22:00:02 +0000 UTC" deadline="2028-06-18 13:08:33.164994587 +0000 UTC" Nov 02 22:05:04.139821 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.139755 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14247h3m29.025242447s" Nov 02 22:05:04.174838 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.174809 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:05:04.190954 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.190899 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:04.251737 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.251706 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:05:04.253840 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.253823 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:05:04.311407 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.311377 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:05:04.432138 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.432104 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:05:04.636890 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:04.636847 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc91a35388f9dd8d39adce6d4d82b0bac.slice/crio-2b5ae94787a66232e1c33984629a0a7e02c459370657d27f42c5cd9b38c17823 WatchSource:0}: Error finding container 2b5ae94787a66232e1c33984629a0a7e02c459370657d27f42c5cd9b38c17823: Status 404 returned error can't find the container with id 2b5ae94787a66232e1c33984629a0a7e02c459370657d27f42c5cd9b38c17823 Nov 02 22:05:04.637415 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:04.637400 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8332e3a50281a723271a56acbbdc782f.slice/crio-777fade67cb5c42aa8ec5cbf133c5f79591fdd26c88efc67ee3689d67ffd47d4 WatchSource:0}: Error finding container 777fade67cb5c42aa8ec5cbf133c5f79591fdd26c88efc67ee3689d67ffd47d4: Status 404 returned error can't find the container with id 777fade67cb5c42aa8ec5cbf133c5f79591fdd26c88efc67ee3689d67ffd47d4 Nov 02 22:05:04.643765 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.643749 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:05:04.656043 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.656024 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:05:04.864270 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.864181 2581 apiserver.go:52] "Watching apiserver" Nov 02 22:05:04.889383 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.889356 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 22:05:04.963067 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.963034 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 22:05:04.963442 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.963420 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal","openshift-image-registry/node-ca-rkr6h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal","kube-system/konnectivity-agent-t2vm8","openshift-dns/node-resolver-vxbxf","openshift-network-diagnostics/network-check-target-92hpx","openshift-ovn-kubernetes/ovnkube-node-hnk92","openshift-cluster-node-tuning-operator/tuned-f94lx","openshift-multus/multus-additional-cni-plugins-4zvx5","openshift-multus/network-metrics-daemon-xfs82","kube-system/global-pull-secret-syncer-tp2df","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd","openshift-multus/multus-sz594","openshift-network-operator/iptables-alerter-mnctf"] Nov 02 22:05:04.966199 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.966175 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:04.967476 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.967456 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sz594" Nov 02 22:05:04.968848 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.968831 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:05:04.970221 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.970204 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:05:04.971635 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.971616 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:05:04.973037 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.972924 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:04.973037 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:04.973002 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:04.974413 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.974396 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:04.976143 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.976123 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:04.977555 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.977536 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:05:04.980479 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.980445 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:04.980609 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:04.980582 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:04.982628 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.982609 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:04.982706 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:04.982673 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:04.984514 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.984497 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:04.992323 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:04.992302 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 22:05:05.003012 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.002984 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.003138 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003019 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-socket-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.003138 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003041 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-system-cni-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.003138 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003063 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8aa26f0e-38c8-4705-b64a-0a943508b307-serviceca\") pod \"node-ca-rkr6h\" (UID: \"8aa26f0e-38c8-4705-b64a-0a943508b307\") " pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:05:05.003138 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003109 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-device-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.003344 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003173 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-kubelet\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.003344 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003200 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwll\" (UniqueName: \"kubernetes.io/projected/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-kube-api-access-qzwll\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.003344 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003224 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysconfig\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.003344 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003249 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-run\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.003344 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003274 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-sys\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.003344 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003316 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf6jx\" (UniqueName: \"kubernetes.io/projected/d725e391-7af5-4691-b327-259e86639f8e-kube-api-access-xf6jx\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.003577 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003354 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-env-overrides\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.003577 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003375 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/425cd6ec-68c1-48ca-8243-b15082ae9553-konnectivity-ca\") pod \"konnectivity-agent-t2vm8\" (UID: \"425cd6ec-68c1-48ca-8243-b15082ae9553\") " pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:05:05.003577 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003399 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be31cb75-89a7-4b84-b26f-2050acfa249c-tmp-dir\") pod \"node-resolver-vxbxf\" (UID: \"be31cb75-89a7-4b84-b26f-2050acfa249c\") " pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:05:05.003577 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003445 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.003577 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003480 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npbhr\" (UniqueName: \"kubernetes.io/projected/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-kube-api-access-npbhr\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.003577 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysctl-conf\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.003577 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003578 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d725e391-7af5-4691-b327-259e86639f8e-etc-tuned\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.003818 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003602 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-etc-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.003818 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003649 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b44243f9-55cd-40b4-87e8-53913b4c26c4-host-slash\") pod \"iptables-alerter-mnctf\" (UID: \"b44243f9-55cd-40b4-87e8-53913b4c26c4\") " pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:05:05.003818 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003692 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cni-binary-copy\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.003818 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003714 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-var-lib-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.003818 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003732 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-node-log\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.003818 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003747 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.003818 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003767 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-multus-certs\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.003818 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003782 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-etc-kubernetes\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.003818 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003797 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-sys-fs\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.003818 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003813 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r6hz\" (UniqueName: \"kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz\") pod \"network-check-target-92hpx\" (UID: \"670c59c6-f81f-47b5-9624-2bab9d1fc847\") " pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:05.004200 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003827 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-system-cni-dir\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.004200 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003841 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-host\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.004200 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003897 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/644947bd-1d54-4a7f-afb6-463500cf17b9-kubelet-config\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:05.004200 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003929 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-cni-binary-copy\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.004200 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.003954 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-conf-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.004200 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004019 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-daemon-config\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.004200 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004038 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-systemd\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.004200 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004070 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-registration-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.004200 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004094 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-cni-bin\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.004200 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004112 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-kubelet\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.004200 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004128 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.004200 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004168 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-k8s-cni-cncf-io\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.004751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004216 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-hostroot\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.004751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004246 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be31cb75-89a7-4b84-b26f-2050acfa249c-hosts-file\") pod \"node-resolver-vxbxf\" (UID: \"be31cb75-89a7-4b84-b26f-2050acfa249c\") " pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:05:05.004751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004272 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.004751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004298 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6g87\" (UniqueName: \"kubernetes.io/projected/8aa26f0e-38c8-4705-b64a-0a943508b307-kube-api-access-m6g87\") pod \"node-ca-rkr6h\" (UID: \"8aa26f0e-38c8-4705-b64a-0a943508b307\") " pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:05:05.004751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004320 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b44243f9-55cd-40b4-87e8-53913b4c26c4-iptables-alerter-script\") pod \"iptables-alerter-mnctf\" (UID: \"b44243f9-55cd-40b4-87e8-53913b4c26c4\") " pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:05:05.004751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004344 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:05.004751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-cni-bin\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.004751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004397 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-etc-selinux\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.004751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004436 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-cnibin\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.004751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004471 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-cni-multus\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.004751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004510 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-modprobe-d\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.004751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004545 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-cni-netd\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.004751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004575 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovn-node-metrics-cert\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.004751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004622 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsphb\" (UniqueName: \"kubernetes.io/projected/2fb97b10-9112-4dd4-8d7d-a2c864c39358-kube-api-access-bsphb\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.004751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004675 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-systemd-units\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.004751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004695 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-ovn\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.005297 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004710 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn52n\" (UniqueName: \"kubernetes.io/projected/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-kube-api-access-pn52n\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.005297 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004729 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8aa26f0e-38c8-4705-b64a-0a943508b307-host\") pod \"node-ca-rkr6h\" (UID: \"8aa26f0e-38c8-4705-b64a-0a943508b307\") " pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:05:05.005297 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004743 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovnkube-config\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.005297 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004773 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwzb\" (UniqueName: \"kubernetes.io/projected/b44243f9-55cd-40b4-87e8-53913b4c26c4-kube-api-access-qgwzb\") pod \"iptables-alerter-mnctf\" (UID: \"b44243f9-55cd-40b4-87e8-53913b4c26c4\") " pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:05:05.005297 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004802 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:05.005297 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004827 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/644947bd-1d54-4a7f-afb6-463500cf17b9-dbus\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:05.005297 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004849 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-cni-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.005297 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004886 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-socket-dir-parent\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.005297 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004931 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-kubernetes\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.005297 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004969 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-run-netns\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.005297 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.004997 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-systemd\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.005297 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.005022 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.005297 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.005078 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz6cr\" (UniqueName: \"kubernetes.io/projected/be31cb75-89a7-4b84-b26f-2050acfa249c-kube-api-access-bz6cr\") pod \"node-resolver-vxbxf\" (UID: \"be31cb75-89a7-4b84-b26f-2050acfa249c\") " pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:05:05.005297 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.005106 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.005297 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.005128 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-slash\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.005297 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.005151 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-log-socket\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.005776 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.005188 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jzj5\" (UniqueName: \"kubernetes.io/projected/6ae0899d-36a2-45b1-9bdf-a76acc087491-kube-api-access-2jzj5\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:05.005776 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.005223 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/425cd6ec-68c1-48ca-8243-b15082ae9553-agent-certs\") pod \"konnectivity-agent-t2vm8\" (UID: \"425cd6ec-68c1-48ca-8243-b15082ae9553\") " pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:05:05.005776 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.005239 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysctl-d\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.005776 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.005255 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-lib-modules\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.005776 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.005278 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-var-lib-kubelet\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.005776 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.005293 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d725e391-7af5-4691-b327-259e86639f8e-tmp\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.005776 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.005306 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovnkube-script-lib\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.005776 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.005329 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-netns\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.005776 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.005357 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cnibin\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.016987 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.016959 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 22:05:05.017110 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.016963 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 22:05:05.017110 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.016959 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 22:05:05.017110 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.017027 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 22:05:05.017110 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.017045 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 22:05:05.017110 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.017084 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-klqgw\"" Nov 02 22:05:05.017346 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.017140 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 22:05:05.017346 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.017243 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 22:05:05.017346 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.017292 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 22:05:05.017346 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.017311 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 22:05:05.017346 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.017312 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rrhfb\"" Nov 02 22:05:05.044092 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.044035 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" event={"ID":"c91a35388f9dd8d39adce6d4d82b0bac","Type":"ContainerStarted","Data":"2b5ae94787a66232e1c33984629a0a7e02c459370657d27f42c5cd9b38c17823"} Nov 02 22:05:05.045120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.045099 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" event={"ID":"8332e3a50281a723271a56acbbdc782f","Type":"ContainerStarted","Data":"777fade67cb5c42aa8ec5cbf133c5f79591fdd26c88efc67ee3689d67ffd47d4"} Nov 02 22:05:05.075320 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.075291 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 22:05:05.075402 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.075339 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4nwct" Nov 02 22:05:05.106362 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106332 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/425cd6ec-68c1-48ca-8243-b15082ae9553-agent-certs\") pod \"konnectivity-agent-t2vm8\" (UID: \"425cd6ec-68c1-48ca-8243-b15082ae9553\") " pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:05:05.106450 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106368 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysctl-d\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.106450 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106384 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-lib-modules\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.106450 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106405 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-var-lib-kubelet\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.106450 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106429 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d725e391-7af5-4691-b327-259e86639f8e-tmp\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.106624 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106457 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovnkube-script-lib\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.106624 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106494 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-var-lib-kubelet\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.106624 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106477 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-netns\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.106624 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106529 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysctl-d\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.106624 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106550 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-lib-modules\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.106624 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106569 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cnibin\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.106852 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106607 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-netns\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.106852 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106680 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cnibin\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.106852 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106693 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.106852 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106726 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-socket-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.106852 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106739 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.106852 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106781 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-system-cni-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.106852 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106826 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8aa26f0e-38c8-4705-b64a-0a943508b307-serviceca\") pod \"node-ca-rkr6h\" (UID: \"8aa26f0e-38c8-4705-b64a-0a943508b307\") " pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:05:05.106852 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106835 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-socket-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.106852 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106850 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-device-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106933 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-device-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106936 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-system-cni-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106939 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106960 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-kubelet\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.106980 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwll\" (UniqueName: \"kubernetes.io/projected/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-kube-api-access-qzwll\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107005 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysconfig\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107016 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-kubelet\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107031 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-run\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107051 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-sys\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107068 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf6jx\" (UniqueName: \"kubernetes.io/projected/d725e391-7af5-4691-b327-259e86639f8e-kube-api-access-xf6jx\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107080 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysconfig\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-env-overrides\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107142 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/425cd6ec-68c1-48ca-8243-b15082ae9553-konnectivity-ca\") pod \"konnectivity-agent-t2vm8\" (UID: \"425cd6ec-68c1-48ca-8243-b15082ae9553\") " pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107147 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-sys\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107189 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-run\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107195 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be31cb75-89a7-4b84-b26f-2050acfa249c-tmp-dir\") pod \"node-resolver-vxbxf\" (UID: \"be31cb75-89a7-4b84-b26f-2050acfa249c\") " pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107209 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107228 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.107304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107247 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npbhr\" (UniqueName: \"kubernetes.io/projected/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-kube-api-access-npbhr\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107263 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysctl-conf\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107335 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d725e391-7af5-4691-b327-259e86639f8e-etc-tuned\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107370 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-etc-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107410 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b44243f9-55cd-40b4-87e8-53913b4c26c4-host-slash\") pod \"iptables-alerter-mnctf\" (UID: \"b44243f9-55cd-40b4-87e8-53913b4c26c4\") " pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107424 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107433 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cni-binary-copy\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-var-lib-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107480 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-node-log\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107535 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107551 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-multus-certs\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107570 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be31cb75-89a7-4b84-b26f-2050acfa249c-tmp-dir\") pod \"node-resolver-vxbxf\" (UID: \"be31cb75-89a7-4b84-b26f-2050acfa249c\") " pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-etc-kubernetes\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107591 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-sys-fs\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107630 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r6hz\" (UniqueName: \"kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz\") pod \"network-check-target-92hpx\" (UID: \"670c59c6-f81f-47b5-9624-2bab9d1fc847\") " pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107645 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-system-cni-dir\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107658 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-host\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.108219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107662 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-etc-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107681 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/644947bd-1d54-4a7f-afb6-463500cf17b9-kubelet-config\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107696 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-cni-binary-copy\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107711 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b44243f9-55cd-40b4-87e8-53913b4c26c4-host-slash\") pod \"iptables-alerter-mnctf\" (UID: \"b44243f9-55cd-40b4-87e8-53913b4c26c4\") " pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107717 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-conf-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107750 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-conf-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107754 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-daemon-config\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107766 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovnkube-script-lib\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107780 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-systemd\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107800 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-registration-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107863 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-cni-bin\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107898 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-kubelet\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107915 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107947 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-k8s-cni-cncf-io\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.107950 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysctl-conf\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108239 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-hostroot\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108313 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-node-log\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108330 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-daemon-config\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.109048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108338 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108404 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-cni-bin\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108419 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/644947bd-1d54-4a7f-afb6-463500cf17b9-kubelet-config\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108455 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-kubelet\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108462 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-multus-certs\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108463 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cni-binary-copy\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108509 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108518 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-registration-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108566 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be31cb75-89a7-4b84-b26f-2050acfa249c-hosts-file\") pod \"node-resolver-vxbxf\" (UID: \"be31cb75-89a7-4b84-b26f-2050acfa249c\") " pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108570 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-etc-kubernetes\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108590 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-systemd\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108600 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108637 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6g87\" (UniqueName: \"kubernetes.io/projected/8aa26f0e-38c8-4705-b64a-0a943508b307-kube-api-access-m6g87\") pod \"node-ca-rkr6h\" (UID: \"8aa26f0e-38c8-4705-b64a-0a943508b307\") " pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108648 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-sys-fs\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108673 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b44243f9-55cd-40b4-87e8-53913b4c26c4-iptables-alerter-script\") pod \"iptables-alerter-mnctf\" (UID: \"b44243f9-55cd-40b4-87e8-53913b4c26c4\") " pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108690 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-var-lib-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108706 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:05.109792 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108738 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-cni-binary-copy\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108785 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-hostroot\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108817 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be31cb75-89a7-4b84-b26f-2050acfa249c-hosts-file\") pod \"node-resolver-vxbxf\" (UID: \"be31cb75-89a7-4b84-b26f-2050acfa249c\") " pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108850 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-k8s-cni-cncf-io\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:05.108891 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108906 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-system-cni-dir\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108965 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-cni-bin\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.108968 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-host\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:05.108998 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret podName:644947bd-1d54-4a7f-afb6-463500cf17b9 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:05.608974146 +0000 UTC m=+3.180840010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret") pod "global-pull-secret-syncer-tp2df" (UID: "644947bd-1d54-4a7f-afb6-463500cf17b9") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.109030 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-cni-bin\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.109062 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-etc-selinux\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.109086 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-cnibin\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.109113 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-cni-multus\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.109136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-modprobe-d\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.109136 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-etc-selinux\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.109172 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-cni-multus\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.109543 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-cnibin\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.110559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.109626 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-cni-netd\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.109765 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-env-overrides\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.109792 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovn-node-metrics-cert\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.109830 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsphb\" (UniqueName: \"kubernetes.io/projected/2fb97b10-9112-4dd4-8d7d-a2c864c39358-kube-api-access-bsphb\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.109926 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-cni-netd\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.109949 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-systemd-units\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.109959 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-modprobe-d\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110012 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-ovn\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110022 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-systemd-units\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110039 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn52n\" (UniqueName: \"kubernetes.io/projected/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-kube-api-access-pn52n\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110077 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-ovn\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110184 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8aa26f0e-38c8-4705-b64a-0a943508b307-host\") pod \"node-ca-rkr6h\" (UID: \"8aa26f0e-38c8-4705-b64a-0a943508b307\") " pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110231 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovnkube-config\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110287 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwzb\" (UniqueName: \"kubernetes.io/projected/b44243f9-55cd-40b4-87e8-53913b4c26c4-kube-api-access-qgwzb\") pod \"iptables-alerter-mnctf\" (UID: \"b44243f9-55cd-40b4-87e8-53913b4c26c4\") " pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110360 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110391 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/644947bd-1d54-4a7f-afb6-463500cf17b9-dbus\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110431 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-cni-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.111347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110514 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-socket-dir-parent\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110546 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-kubernetes\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110578 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-run-netns\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110609 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-systemd\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110660 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/644947bd-1d54-4a7f-afb6-463500cf17b9-dbus\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110681 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-cni-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110669 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110742 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz6cr\" (UniqueName: \"kubernetes.io/projected/be31cb75-89a7-4b84-b26f-2050acfa249c-kube-api-access-bz6cr\") pod \"node-resolver-vxbxf\" (UID: \"be31cb75-89a7-4b84-b26f-2050acfa249c\") " pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110789 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110802 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110817 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-slash\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110839 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-run-netns\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110851 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-log-socket\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.110902 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzj5\" (UniqueName: \"kubernetes.io/projected/6ae0899d-36a2-45b1-9bdf-a76acc087491-kube-api-access-2jzj5\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:05.110912 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:05.110955 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs podName:6ae0899d-36a2-45b1-9bdf-a76acc087491 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:05.610944589 +0000 UTC m=+3.182810453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs") pod "network-metrics-daemon-xfs82" (UID: "6ae0899d-36a2-45b1-9bdf-a76acc087491") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.111036 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-systemd\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.112120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.111336 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovnkube-config\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.112832 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.111362 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-slash\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.112832 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.111434 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-kubernetes\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.112832 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.111448 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-socket-dir-parent\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.112832 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.111469 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-log-socket\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.112832 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.111363 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d725e391-7af5-4691-b327-259e86639f8e-etc-tuned\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.112832 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.111586 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8aa26f0e-38c8-4705-b64a-0a943508b307-host\") pod \"node-ca-rkr6h\" (UID: \"8aa26f0e-38c8-4705-b64a-0a943508b307\") " pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:05:05.112832 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.111783 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d725e391-7af5-4691-b327-259e86639f8e-tmp\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.114157 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.114134 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovn-node-metrics-cert\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.117440 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.117395 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 22:05:05.117695 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.117678 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 22:05:05.117954 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.117937 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 22:05:05.118192 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.118179 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 22:05:05.118192 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.118184 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 22:05:05.122960 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.122939 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:05:05.127646 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.127615 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8aa26f0e-38c8-4705-b64a-0a943508b307-serviceca\") pod \"node-ca-rkr6h\" (UID: \"8aa26f0e-38c8-4705-b64a-0a943508b307\") " pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:05:05.136688 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136656 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 22:05:05.136688 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136679 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 22:05:05.136688 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136692 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 22:05:05.136688 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136679 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 22:05:05.137026 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136684 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 22:05:05.137026 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136675 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:05:05.137026 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136719 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 22:05:05.137026 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136720 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 22:05:05.137026 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136755 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 22:05:05.137026 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136762 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rksbh\"" Nov 02 22:05:05.137026 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136767 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rsfpb\"" Nov 02 22:05:05.137026 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136774 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fcg7g\"" Nov 02 22:05:05.137026 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136786 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 22:05:05.137026 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136790 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4ts5p\"" Nov 02 22:05:05.137026 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136780 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 22:05:05.137026 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136807 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4rxp8\"" Nov 02 22:05:05.137026 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136853 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 22:05:05.137026 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136922 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zcntw\"" Nov 02 22:05:05.137026 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.136967 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-656w2\"" Nov 02 22:05:05.138310 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.138289 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/425cd6ec-68c1-48ca-8243-b15082ae9553-konnectivity-ca\") pod \"konnectivity-agent-t2vm8\" (UID: \"425cd6ec-68c1-48ca-8243-b15082ae9553\") " pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:05:05.139399 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.139381 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b44243f9-55cd-40b4-87e8-53913b4c26c4-iptables-alerter-script\") pod \"iptables-alerter-mnctf\" (UID: \"b44243f9-55cd-40b4-87e8-53913b4c26c4\") " pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:05:05.139444 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.139403 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.142082 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.142062 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.149284 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.149258 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/425cd6ec-68c1-48ca-8243-b15082ae9553-agent-certs\") pod \"konnectivity-agent-t2vm8\" (UID: \"425cd6ec-68c1-48ca-8243-b15082ae9553\") " pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:05:05.165021 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:05.164997 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:05:05.165021 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:05.165021 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:05:05.165130 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:05.165033 2581 projected.go:196] Error preparing data for projected volume kube-api-access-4r6hz for pod openshift-network-diagnostics/network-check-target-92hpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:05.165130 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:05.165100 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz podName:670c59c6-f81f-47b5-9624-2bab9d1fc847 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:05.665079132 +0000 UTC m=+3.236945017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4r6hz" (UniqueName: "kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz") pod "network-check-target-92hpx" (UID: "670c59c6-f81f-47b5-9624-2bab9d1fc847") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:05.167447 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.167424 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6g87\" (UniqueName: \"kubernetes.io/projected/8aa26f0e-38c8-4705-b64a-0a943508b307-kube-api-access-m6g87\") pod \"node-ca-rkr6h\" (UID: \"8aa26f0e-38c8-4705-b64a-0a943508b307\") " pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:05:05.167557 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.167472 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf6jx\" (UniqueName: \"kubernetes.io/projected/d725e391-7af5-4691-b327-259e86639f8e-kube-api-access-xf6jx\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.167557 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.167539 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwll\" (UniqueName: \"kubernetes.io/projected/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-kube-api-access-qzwll\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:05:05.167625 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.167593 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npbhr\" (UniqueName: \"kubernetes.io/projected/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-kube-api-access-npbhr\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.168344 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.168323 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn52n\" (UniqueName: \"kubernetes.io/projected/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-kube-api-access-pn52n\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.169911 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.169892 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwzb\" (UniqueName: \"kubernetes.io/projected/b44243f9-55cd-40b4-87e8-53913b4c26c4-kube-api-access-qgwzb\") pod \"iptables-alerter-mnctf\" (UID: \"b44243f9-55cd-40b4-87e8-53913b4c26c4\") " pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:05:05.174987 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.174963 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz6cr\" (UniqueName: \"kubernetes.io/projected/be31cb75-89a7-4b84-b26f-2050acfa249c-kube-api-access-bz6cr\") pod \"node-resolver-vxbxf\" (UID: \"be31cb75-89a7-4b84-b26f-2050acfa249c\") " pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:05:05.175064 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.175026 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsphb\" (UniqueName: \"kubernetes.io/projected/2fb97b10-9112-4dd4-8d7d-a2c864c39358-kube-api-access-bsphb\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.175064 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.175053 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jzj5\" (UniqueName: \"kubernetes.io/projected/6ae0899d-36a2-45b1-9bdf-a76acc087491-kube-api-access-2jzj5\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:05.286449 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.286413 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:05.287453 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.287430 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sz594" Nov 02 22:05:05.292855 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.292823 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:05:05.296763 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.296740 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:05:05.298534 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:05.298507 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb97b10_9112_4dd4_8d7d_a2c864c39358.slice/crio-a72e21182b1155c0e19ae1737ceb9e5537968ab3a3246bd4dbe19a5c590214ed WatchSource:0}: Error finding container a72e21182b1155c0e19ae1737ceb9e5537968ab3a3246bd4dbe19a5c590214ed: Status 404 returned error can't find the container with id a72e21182b1155c0e19ae1737ceb9e5537968ab3a3246bd4dbe19a5c590214ed Nov 02 22:05:05.301921 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:05.301892 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6652ca_37d9_45f8_a135_19dc4cc60fc3.slice/crio-d11690e1d9b5e4668765d0d4c930146f950b078d877671ac7e94053623d6c028 WatchSource:0}: Error finding container d11690e1d9b5e4668765d0d4c930146f950b078d877671ac7e94053623d6c028: Status 404 returned error can't find the container with id d11690e1d9b5e4668765d0d4c930146f950b078d877671ac7e94053623d6c028 Nov 02 22:05:05.302557 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.302448 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:05:05.306014 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:05.305987 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44243f9_55cd_40b4_87e8_53913b4c26c4.slice/crio-c750bbf4e947a072895ce3eb34c9a418999b007f53c73d462c9cc8ad410dfff5 WatchSource:0}: Error finding container c750bbf4e947a072895ce3eb34c9a418999b007f53c73d462c9cc8ad410dfff5: Status 404 returned error can't find the container with id c750bbf4e947a072895ce3eb34c9a418999b007f53c73d462c9cc8ad410dfff5 Nov 02 22:05:05.309479 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:05.309456 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425cd6ec_68c1_48ca_8243_b15082ae9553.slice/crio-30d3ea3093c8ab907dacae103f38bbc7f21c76d106337a775fe667dbba6f670c WatchSource:0}: Error finding container 30d3ea3093c8ab907dacae103f38bbc7f21c76d106337a775fe667dbba6f670c: Status 404 returned error can't find the container with id 30d3ea3093c8ab907dacae103f38bbc7f21c76d106337a775fe667dbba6f670c Nov 02 22:05:05.312447 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:05.312396 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe31cb75_89a7_4b84_b26f_2050acfa249c.slice/crio-794a5fcae42ade03d3565c050f435fdfc9f71feaa020200de7b8456b89148471 WatchSource:0}: Error finding container 794a5fcae42ade03d3565c050f435fdfc9f71feaa020200de7b8456b89148471: Status 404 returned error can't find the container with id 794a5fcae42ade03d3565c050f435fdfc9f71feaa020200de7b8456b89148471 Nov 02 22:05:05.316574 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.316550 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:05:05.317640 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.317617 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:05:05.321445 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.321423 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:05:05.326791 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:05.326766 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e3a7cc_c4c9_44d7_b358_029b6a8c9cc3.slice/crio-90ce1faee2a613e6e6b1dd0ffa46235eb09b18377f9bf507c45d583f3efff9ed WatchSource:0}: Error finding container 90ce1faee2a613e6e6b1dd0ffa46235eb09b18377f9bf507c45d583f3efff9ed: Status 404 returned error can't find the container with id 90ce1faee2a613e6e6b1dd0ffa46235eb09b18377f9bf507c45d583f3efff9ed Nov 02 22:05:05.327475 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.327438 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:05:05.327543 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:05.327521 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd725e391_7af5_4691_b327_259e86639f8e.slice/crio-bab61dd34dc3c437a113521a086eecfd2d1e1e59f3421345655388fee1230854 WatchSource:0}: Error finding container bab61dd34dc3c437a113521a086eecfd2d1e1e59f3421345655388fee1230854: Status 404 returned error can't find the container with id bab61dd34dc3c437a113521a086eecfd2d1e1e59f3421345655388fee1230854 Nov 02 22:05:05.330762 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:05.330738 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa26f0e_38c8_4705_b64a_0a943508b307.slice/crio-9d10e0153a84c93325ad46da510a95e83f03b8e16b45a623938c384724c6bb8e WatchSource:0}: Error finding container 9d10e0153a84c93325ad46da510a95e83f03b8e16b45a623938c384724c6bb8e: Status 404 returned error can't find the container with id 9d10e0153a84c93325ad46da510a95e83f03b8e16b45a623938c384724c6bb8e Nov 02 22:05:05.336239 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:05.336212 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8a9a43_5fd2_4e49_b313_9ff58be7524b.slice/crio-b0ff9b530e22aee46bd8c080adf8af0cc217c8634220388e791d49352153bf1f WatchSource:0}: Error finding container b0ff9b530e22aee46bd8c080adf8af0cc217c8634220388e791d49352153bf1f: Status 404 returned error can't find the container with id b0ff9b530e22aee46bd8c080adf8af0cc217c8634220388e791d49352153bf1f Nov 02 22:05:05.615032 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.614947 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:05.615032 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.614989 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:05.615215 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:05.615106 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:05.615215 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:05.615165 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs podName:6ae0899d-36a2-45b1-9bdf-a76acc087491 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:06.615152162 +0000 UTC m=+4.187018040 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs") pod "network-metrics-daemon-xfs82" (UID: "6ae0899d-36a2-45b1-9bdf-a76acc087491") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:05.615215 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:05.615106 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:05.615215 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:05.615204 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret podName:644947bd-1d54-4a7f-afb6-463500cf17b9 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:06.61519831 +0000 UTC m=+4.187064174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret") pod "global-pull-secret-syncer-tp2df" (UID: "644947bd-1d54-4a7f-afb6-463500cf17b9") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:05.715540 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:05.715498 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r6hz\" (UniqueName: \"kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz\") pod \"network-check-target-92hpx\" (UID: \"670c59c6-f81f-47b5-9624-2bab9d1fc847\") " pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:05.715710 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:05.715657 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:05:05.715710 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:05.715677 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:05:05.715710 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:05.715689 2581 projected.go:196] Error preparing data for projected volume kube-api-access-4r6hz for pod openshift-network-diagnostics/network-check-target-92hpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:05.715853 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:05.715759 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz podName:670c59c6-f81f-47b5-9624-2bab9d1fc847 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:06.715740186 +0000 UTC m=+4.287606070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4r6hz" (UniqueName: "kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz") pod "network-check-target-92hpx" (UID: "670c59c6-f81f-47b5-9624-2bab9d1fc847") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:06.048329 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:06.048241 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f94lx" event={"ID":"d725e391-7af5-4691-b327-259e86639f8e","Type":"ContainerStarted","Data":"bab61dd34dc3c437a113521a086eecfd2d1e1e59f3421345655388fee1230854"} Nov 02 22:05:06.049800 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:06.049762 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zvx5" event={"ID":"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3","Type":"ContainerStarted","Data":"90ce1faee2a613e6e6b1dd0ffa46235eb09b18377f9bf507c45d583f3efff9ed"} Nov 02 22:05:06.051108 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:06.051035 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mnctf" event={"ID":"b44243f9-55cd-40b4-87e8-53913b4c26c4","Type":"ContainerStarted","Data":"c750bbf4e947a072895ce3eb34c9a418999b007f53c73d462c9cc8ad410dfff5"} Nov 02 22:05:06.052935 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:06.052900 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" event={"ID":"8b8a9a43-5fd2-4e49-b313-9ff58be7524b","Type":"ContainerStarted","Data":"b0ff9b530e22aee46bd8c080adf8af0cc217c8634220388e791d49352153bf1f"} Nov 02 22:05:06.058125 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:06.058077 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rkr6h" event={"ID":"8aa26f0e-38c8-4705-b64a-0a943508b307","Type":"ContainerStarted","Data":"9d10e0153a84c93325ad46da510a95e83f03b8e16b45a623938c384724c6bb8e"} Nov 02 22:05:06.059602 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:06.059557 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vxbxf" event={"ID":"be31cb75-89a7-4b84-b26f-2050acfa249c","Type":"ContainerStarted","Data":"794a5fcae42ade03d3565c050f435fdfc9f71feaa020200de7b8456b89148471"} Nov 02 22:05:06.061005 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:06.060974 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t2vm8" event={"ID":"425cd6ec-68c1-48ca-8243-b15082ae9553","Type":"ContainerStarted","Data":"30d3ea3093c8ab907dacae103f38bbc7f21c76d106337a775fe667dbba6f670c"} Nov 02 22:05:06.062514 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:06.062483 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sz594" event={"ID":"2f6652ca-37d9-45f8-a135-19dc4cc60fc3","Type":"ContainerStarted","Data":"d11690e1d9b5e4668765d0d4c930146f950b078d877671ac7e94053623d6c028"} Nov 02 22:05:06.064081 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:06.064049 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" event={"ID":"2fb97b10-9112-4dd4-8d7d-a2c864c39358","Type":"ContainerStarted","Data":"a72e21182b1155c0e19ae1737ceb9e5537968ab3a3246bd4dbe19a5c590214ed"} Nov 02 22:05:06.078084 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:06.078025 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:00:04 +0000 UTC" deadline="2028-05-22 03:55:36.432975649 +0000 UTC" Nov 02 22:05:06.078084 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:06.078080 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13589h50m30.354899175s" Nov 02 22:05:06.624084 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:06.624038 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:06.624279 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:06.624104 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:06.624279 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:06.624238 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:06.624399 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:06.624301 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs podName:6ae0899d-36a2-45b1-9bdf-a76acc087491 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:08.624282846 +0000 UTC m=+6.196148725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs") pod "network-metrics-daemon-xfs82" (UID: "6ae0899d-36a2-45b1-9bdf-a76acc087491") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:06.624399 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:06.624363 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:06.624399 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:06.624396 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret podName:644947bd-1d54-4a7f-afb6-463500cf17b9 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:08.624385428 +0000 UTC m=+6.196251293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret") pod "global-pull-secret-syncer-tp2df" (UID: "644947bd-1d54-4a7f-afb6-463500cf17b9") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:06.724941 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:06.724868 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r6hz\" (UniqueName: \"kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz\") pod \"network-check-target-92hpx\" (UID: \"670c59c6-f81f-47b5-9624-2bab9d1fc847\") " pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:06.725192 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:06.725062 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:05:06.725192 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:06.725082 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:05:06.725192 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:06.725095 2581 projected.go:196] Error preparing data for projected volume kube-api-access-4r6hz for pod openshift-network-diagnostics/network-check-target-92hpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:06.725192 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:06.725155 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz podName:670c59c6-f81f-47b5-9624-2bab9d1fc847 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:08.72513817 +0000 UTC m=+6.297004048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4r6hz" (UniqueName: "kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz") pod "network-check-target-92hpx" (UID: "670c59c6-f81f-47b5-9624-2bab9d1fc847") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:07.043137 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:07.043055 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:07.043293 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:07.043201 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:07.043359 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:07.043301 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:07.043418 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:07.043382 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:07.043479 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:07.043442 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:07.043742 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:07.043522 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:07.078625 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:07.078533 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:00:04 +0000 UTC" deadline="2028-04-09 21:04:55.497692548 +0000 UTC" Nov 02 22:05:07.078625 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:07.078594 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12574h59m48.419103353s" Nov 02 22:05:08.640764 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:08.640402 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:08.640764 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:08.640465 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:08.640764 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:08.640595 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:08.640764 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:08.640651 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs podName:6ae0899d-36a2-45b1-9bdf-a76acc087491 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:12.640631832 +0000 UTC m=+10.212497696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs") pod "network-metrics-daemon-xfs82" (UID: "6ae0899d-36a2-45b1-9bdf-a76acc087491") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:08.640764 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:08.640702 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:08.640764 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:08.640728 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret podName:644947bd-1d54-4a7f-afb6-463500cf17b9 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:12.64072105 +0000 UTC m=+10.212586914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret") pod "global-pull-secret-syncer-tp2df" (UID: "644947bd-1d54-4a7f-afb6-463500cf17b9") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:08.742138 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:08.741643 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r6hz\" (UniqueName: \"kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz\") pod \"network-check-target-92hpx\" (UID: \"670c59c6-f81f-47b5-9624-2bab9d1fc847\") " pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:08.742138 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:08.741984 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:05:08.742138 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:08.742006 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:05:08.742138 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:08.742019 2581 projected.go:196] Error preparing data for projected volume kube-api-access-4r6hz for pod openshift-network-diagnostics/network-check-target-92hpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:08.742138 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:08.742094 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz podName:670c59c6-f81f-47b5-9624-2bab9d1fc847 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:12.742075743 +0000 UTC m=+10.313941622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4r6hz" (UniqueName: "kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz") pod "network-check-target-92hpx" (UID: "670c59c6-f81f-47b5-9624-2bab9d1fc847") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:09.043086 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:09.042500 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:09.043086 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:09.042641 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:09.043086 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:09.042709 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:09.043086 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:09.042798 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:09.043086 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:09.042846 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:09.043086 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:09.042935 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:09.107610 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:09.107415 2581 generic.go:356] "Generic (PLEG): container finished" podID="c91a35388f9dd8d39adce6d4d82b0bac" containerID="d2f7cdaaeea9512c6b237657365664e2bed7a5e85435fd140d7ead330646aa22" exitCode=0 Nov 02 22:05:09.114502 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:09.114453 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" event={"ID":"c91a35388f9dd8d39adce6d4d82b0bac","Type":"ContainerDied","Data":"d2f7cdaaeea9512c6b237657365664e2bed7a5e85435fd140d7ead330646aa22"} Nov 02 22:05:09.124495 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:09.124416 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" event={"ID":"8332e3a50281a723271a56acbbdc782f","Type":"ContainerStarted","Data":"826c346915a554ac42e7d8030e5969ffb37e500d4b980df7b58014fd15972282"} Nov 02 22:05:09.132660 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:09.131423 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zvx5" event={"ID":"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3","Type":"ContainerStarted","Data":"e39358ebbd94827a980d1949af3d31fd9358d9faec3ab07773a4e04451e09fa6"} Nov 02 22:05:10.158501 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:10.158458 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" event={"ID":"c91a35388f9dd8d39adce6d4d82b0bac","Type":"ContainerStarted","Data":"67cc4224298aabcc53ae23f53cc40e031891b0ac62c4363297e5f9a8828667c6"} Nov 02 22:05:10.162917 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:10.162887 2581 generic.go:356] "Generic (PLEG): container finished" podID="44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3" containerID="e39358ebbd94827a980d1949af3d31fd9358d9faec3ab07773a4e04451e09fa6" exitCode=0 Nov 02 22:05:10.163821 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:10.163792 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zvx5" event={"ID":"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3","Type":"ContainerDied","Data":"e39358ebbd94827a980d1949af3d31fd9358d9faec3ab07773a4e04451e09fa6"} Nov 02 22:05:10.271463 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:10.271393 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" podStartSLOduration=6.271376602 podStartE2EDuration="6.271376602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:09.458863968 +0000 UTC m=+7.030729854" watchObservedRunningTime="2026-11-02 22:05:10.271376602 +0000 UTC m=+7.843242488" Nov 02 22:05:10.271656 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:10.271570 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" podStartSLOduration=6.271560537 podStartE2EDuration="6.271560537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:10.270992397 +0000 UTC m=+7.842858284" watchObservedRunningTime="2026-11-02 22:05:10.271560537 +0000 UTC m=+7.843426423" Nov 02 22:05:11.042722 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:11.042688 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:11.042911 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:11.042830 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:11.042961 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:11.042931 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:11.043057 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:11.043025 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:11.043130 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:11.043096 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:11.043202 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:11.043174 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:11.169066 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:11.169030 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t2vm8" event={"ID":"425cd6ec-68c1-48ca-8243-b15082ae9553","Type":"ContainerStarted","Data":"c985880a7409cf4d81495101761808aa3336df108bde58f636519693166a94c5"} Nov 02 22:05:12.174697 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:12.174617 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" event={"ID":"8b8a9a43-5fd2-4e49-b313-9ff58be7524b","Type":"ContainerStarted","Data":"dc829d6af1d6d9576c50e68b3c88d19aa9773e32d627ccffdb900704ee077796"} Nov 02 22:05:12.179157 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:12.179120 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rkr6h" event={"ID":"8aa26f0e-38c8-4705-b64a-0a943508b307","Type":"ContainerStarted","Data":"f8837d2a407de44e548c4922a0b3f873dd854d1174f7bef90b5bc57bb4e17c48"} Nov 02 22:05:12.289672 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:12.289560 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rkr6h" podStartSLOduration=3.535900519 podStartE2EDuration="9.289541804s" totalImagesPullingTime="5.753641285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:12.289277541 +0000 UTC m=+9.861143428" watchObservedRunningTime="2026-11-02 22:05:12.289541804 +0000 UTC m=+9.861407692" Nov 02 22:05:12.289825 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:12.289715 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t2vm8" podStartSLOduration=4.550192624 podStartE2EDuration="9.289707617s" totalImagesPullingTime="4.739514993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:11.280462225 +0000 UTC m=+8.852328111" watchObservedRunningTime="2026-11-02 22:05:12.289707617 +0000 UTC m=+9.861573506" Nov 02 22:05:12.626567 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:12.626488 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:05:12.627237 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:12.627212 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:05:12.674832 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:12.674786 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:12.675020 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:12.674852 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:12.675080 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:12.675021 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:12.675147 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:12.675114 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs podName:6ae0899d-36a2-45b1-9bdf-a76acc087491 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:20.67509551 +0000 UTC m=+18.246961391 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs") pod "network-metrics-daemon-xfs82" (UID: "6ae0899d-36a2-45b1-9bdf-a76acc087491") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:12.675205 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:12.675188 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:12.675264 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:12.675225 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret podName:644947bd-1d54-4a7f-afb6-463500cf17b9 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:20.675214446 +0000 UTC m=+18.247080313 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret") pod "global-pull-secret-syncer-tp2df" (UID: "644947bd-1d54-4a7f-afb6-463500cf17b9") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:12.775857 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:12.775817 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r6hz\" (UniqueName: \"kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz\") pod \"network-check-target-92hpx\" (UID: \"670c59c6-f81f-47b5-9624-2bab9d1fc847\") " pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:12.776037 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:12.776006 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:05:12.776037 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:12.776026 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:05:12.776154 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:12.776039 2581 projected.go:196] Error preparing data for projected volume kube-api-access-4r6hz for pod openshift-network-diagnostics/network-check-target-92hpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:12.776154 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:12.776100 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz podName:670c59c6-f81f-47b5-9624-2bab9d1fc847 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:20.776082709 +0000 UTC m=+18.347948576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4r6hz" (UniqueName: "kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz") pod "network-check-target-92hpx" (UID: "670c59c6-f81f-47b5-9624-2bab9d1fc847") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:13.044914 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:13.044296 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:13.044914 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:13.044423 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:13.044914 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:13.044513 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:13.044914 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:13.044585 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:13.044914 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:13.044642 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:13.044914 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:13.044709 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:14.082478 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:14.082440 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 22:05:14.189479 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:14.189387 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sz594" event={"ID":"2f6652ca-37d9-45f8-a135-19dc4cc60fc3","Type":"ContainerStarted","Data":"b8bb330ec7b64a65786adf8a38c1204cd1be37c918c48ab878d550e02e3e2a87"} Nov 02 22:05:14.194474 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:14.194445 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:05:14.194950 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:14.194926 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" event={"ID":"8b8a9a43-5fd2-4e49-b313-9ff58be7524b","Type":"ContainerStarted","Data":"11b5abd7192bf60e9df2f1ad08e9100166b2909175036fe63a2dc330edc15cb7"} Nov 02 22:05:14.974273 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:14.973890 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T22:05:14.082470112Z","UUID":"bcd7d00b-9a85-4c40-80c1-74037e8865e8","Handler":null,"Name":"","Endpoint":""} Nov 02 22:05:14.976022 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:14.975556 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 22:05:14.976022 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:14.975589 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 22:05:15.042093 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:15.042052 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:15.042283 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:15.042182 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:15.042283 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:15.042266 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:15.042357 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:15.042337 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:15.042409 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:15.042390 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:15.042502 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:15.042485 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:15.201936 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:15.201140 2581 generic.go:356] "Generic (PLEG): container finished" podID="44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3" containerID="014bbf9e271fa69c5ab298a2c7e15af22906259b557ea37be8256917f139c903" exitCode=0 Nov 02 22:05:15.201936 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:15.201736 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zvx5" event={"ID":"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3","Type":"ContainerDied","Data":"014bbf9e271fa69c5ab298a2c7e15af22906259b557ea37be8256917f139c903"} Nov 02 22:05:15.205993 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:15.205938 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vxbxf" event={"ID":"be31cb75-89a7-4b84-b26f-2050acfa249c","Type":"ContainerStarted","Data":"8efe54f3a6b2c1555859f6878fde5973fe732c0cd233fa6b4dc048afa562bd21"} Nov 02 22:05:15.320587 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:15.320483 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-sz594" podStartSLOduration=4.036803338 podStartE2EDuration="12.320463793s" totalImagesPullingTime="8.283660455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:14.303746995 +0000 UTC m=+11.875612875" watchObservedRunningTime="2026-11-02 22:05:15.320463793 +0000 UTC m=+12.892329680" Nov 02 22:05:15.430108 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:15.430051 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-vxbxf" podStartSLOduration=3.625073841 podStartE2EDuration="12.430036678s" totalImagesPullingTime="8.804962837s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:15.42993692 +0000 UTC m=+13.001802808" watchObservedRunningTime="2026-11-02 22:05:15.430036678 +0000 UTC m=+13.001902565" Nov 02 22:05:16.209684 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:16.209641 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" event={"ID":"8b8a9a43-5fd2-4e49-b313-9ff58be7524b","Type":"ContainerStarted","Data":"14fc84ab1d4c2102a14e1f72dd538bd9c9ce3a90e652426974a9b3e4003f2d0f"} Nov 02 22:05:16.211329 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:16.211294 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mnctf" event={"ID":"b44243f9-55cd-40b4-87e8-53913b4c26c4","Type":"ContainerStarted","Data":"6ae9563f9a8e05fa5df52523dcf47d827f2c6243587d1ae8d660606baacd1552"} Nov 02 22:05:16.325729 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:16.325623 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" podStartSLOduration=3.420268784 podStartE2EDuration="13.325609318s" totalImagesPullingTime="9.905340534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:05:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:16.325018584 +0000 UTC m=+13.896884471" watchObservedRunningTime="2026-11-02 22:05:16.325609318 +0000 UTC m=+13.897475204" Nov 02 22:05:16.435287 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:16.435217 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mnctf" podStartSLOduration=4.471485124 podStartE2EDuration="13.435196487s" totalImagesPullingTime="8.963711363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:16.43456283 +0000 UTC m=+14.006428723" watchObservedRunningTime="2026-11-02 22:05:16.435196487 +0000 UTC m=+14.007062376" Nov 02 22:05:17.042313 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:17.042281 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:17.042470 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:17.042419 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:17.042532 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:17.042495 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:17.042596 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:17.042574 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:17.042654 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:17.042631 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:17.042719 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:17.042700 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:17.215539 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:17.215449 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f94lx" event={"ID":"d725e391-7af5-4691-b327-259e86639f8e","Type":"ContainerStarted","Data":"8a72fe276709616f746352a1325196222f603a58ddc740b55b50e6835aff68fa"} Nov 02 22:05:17.217987 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:17.217957 2581 generic.go:356] "Generic (PLEG): container finished" podID="44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3" containerID="8b860c812745c401ce78678f8c1f82cad9c0ce297d3af926f4047e45d2f43897" exitCode=0 Nov 02 22:05:17.218284 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:17.218253 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zvx5" event={"ID":"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3","Type":"ContainerDied","Data":"8b860c812745c401ce78678f8c1f82cad9c0ce297d3af926f4047e45d2f43897"} Nov 02 22:05:17.330864 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:17.330811 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f94lx" podStartSLOduration=3.123070782 podStartE2EDuration="14.330797334s" totalImagesPullingTime="11.207726552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:17.330324884 +0000 UTC m=+14.902190771" watchObservedRunningTime="2026-11-02 22:05:17.330797334 +0000 UTC m=+14.902663220" Nov 02 22:05:19.029405 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:19.029368 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:19.040416 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:19.040385 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:19.040584 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:19.040380 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:19.040584 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:19.040521 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:19.040584 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:19.040528 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:19.040719 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:19.040624 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:19.040758 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:19.040720 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:19.225993 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:19.225908 2581 generic.go:356] "Generic (PLEG): container finished" podID="44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3" containerID="57bf7e0c1cc20fabf4068af3e5672f02c0aa7a8176abe3c905486cda5c1588bc" exitCode=0 Nov 02 22:05:19.225993 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:19.225956 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zvx5" event={"ID":"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3","Type":"ContainerDied","Data":"57bf7e0c1cc20fabf4068af3e5672f02c0aa7a8176abe3c905486cda5c1588bc"} Nov 02 22:05:19.261158 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:19.261123 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:20.599691 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:20.599659 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:20.742139 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:20.742103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:20.742287 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:20.742163 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:20.742287 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:20.742248 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:20.742287 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:20.742252 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:20.742400 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:20.742300 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret podName:644947bd-1d54-4a7f-afb6-463500cf17b9 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:36.742287793 +0000 UTC m=+34.314153656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret") pod "global-pull-secret-syncer-tp2df" (UID: "644947bd-1d54-4a7f-afb6-463500cf17b9") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:20.742400 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:20.742313 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs podName:6ae0899d-36a2-45b1-9bdf-a76acc087491 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:36.742306331 +0000 UTC m=+34.314172196 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs") pod "network-metrics-daemon-xfs82" (UID: "6ae0899d-36a2-45b1-9bdf-a76acc087491") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:20.776882 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:20.776836 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:20.842610 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:20.842582 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r6hz\" (UniqueName: \"kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz\") pod \"network-check-target-92hpx\" (UID: \"670c59c6-f81f-47b5-9624-2bab9d1fc847\") " pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:20.842769 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:20.842749 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:05:20.842845 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:20.842772 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:05:20.842845 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:20.842787 2581 projected.go:196] Error preparing data for projected volume kube-api-access-4r6hz for pod openshift-network-diagnostics/network-check-target-92hpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:20.842845 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:20.842839 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz podName:670c59c6-f81f-47b5-9624-2bab9d1fc847 nodeName:}" failed. No retries permitted until 2026-11-02 22:05:36.842826581 +0000 UTC m=+34.414692449 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4r6hz" (UniqueName: "kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz") pod "network-check-target-92hpx" (UID: "670c59c6-f81f-47b5-9624-2bab9d1fc847") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:21.041022 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:21.040833 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:21.041165 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:21.040833 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:21.041165 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:21.040857 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:21.041803 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:21.041771 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:21.041988 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:21.041969 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:21.042098 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:21.042078 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:21.233068 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:21.232981 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" event={"ID":"2fb97b10-9112-4dd4-8d7d-a2c864c39358","Type":"ContainerStarted","Data":"bcb94e61bcac3145bd97007a8b015551bba49592b80a8bb0d9e94da11f93b05b"} Nov 02 22:05:21.233068 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:21.233058 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" event={"ID":"2fb97b10-9112-4dd4-8d7d-a2c864c39358","Type":"ContainerStarted","Data":"6017535c20dba65b5558209d41c8e0bb0f35930ccd992281bcd34a0787fb49f8"} Nov 02 22:05:21.233255 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:21.233095 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" event={"ID":"2fb97b10-9112-4dd4-8d7d-a2c864c39358","Type":"ContainerStarted","Data":"f07f35601c55d0c828f14c3831bfcb15c93cc1c848fa7a2adbdc8a2036ffb17e"} Nov 02 22:05:21.233255 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:21.233123 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" event={"ID":"2fb97b10-9112-4dd4-8d7d-a2c864c39358","Type":"ContainerStarted","Data":"84d4f13572c437727714112a158efa86028a8cf10d8b256837c111c9fd2f8e0e"} Nov 02 22:05:21.233255 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:21.233149 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" event={"ID":"2fb97b10-9112-4dd4-8d7d-a2c864c39358","Type":"ContainerStarted","Data":"bbdc36563abe3f123069a22b6a7364bc0e4e476f4f732a41cc0d9785a069e7ac"} Nov 02 22:05:21.233255 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:21.233186 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" event={"ID":"2fb97b10-9112-4dd4-8d7d-a2c864c39358","Type":"ContainerStarted","Data":"f1c7052880a4ba8e6f6ca0acb35f916a8c2bcedde91d3de327fafd78036a1504"} Nov 02 22:05:23.041122 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:23.041085 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:23.041656 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:23.041185 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:23.041656 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:23.041258 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:23.041656 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:23.041355 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:23.041656 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:23.041393 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:23.041656 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:23.041439 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:23.240441 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:23.240405 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" event={"ID":"2fb97b10-9112-4dd4-8d7d-a2c864c39358","Type":"ContainerStarted","Data":"307088b0d1db04c49ec077af3c529fcc6db4548a9afe210f8905b23672d4b88c"} Nov 02 22:05:23.246679 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:23.246641 2581 generic.go:356] "Generic (PLEG): container finished" podID="44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3" containerID="5f973736c678ae54333a86eccc8c51102a1c7cfc53959c9692a391020fb50c12" exitCode=0 Nov 02 22:05:23.246825 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:23.246694 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zvx5" event={"ID":"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3","Type":"ContainerDied","Data":"5f973736c678ae54333a86eccc8c51102a1c7cfc53959c9692a391020fb50c12"} Nov 02 22:05:24.152450 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:24.152415 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:24.251495 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:24.251458 2581 generic.go:356] "Generic (PLEG): container finished" podID="44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3" containerID="5967fcec3a0871fdda577534ce21d58fd8cfbbe282c38b64c0e7e2e658e34d9b" exitCode=0 Nov 02 22:05:24.251646 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:24.251535 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zvx5" event={"ID":"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3","Type":"ContainerDied","Data":"5967fcec3a0871fdda577534ce21d58fd8cfbbe282c38b64c0e7e2e658e34d9b"} Nov 02 22:05:24.328543 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:24.328518 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:24.519089 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:24.519010 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:24.702702 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:24.702667 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:24.884206 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:24.884175 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:25.040818 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.040779 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:25.040818 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.040804 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:25.041073 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:25.040904 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:25.041073 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.040949 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:25.041073 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:25.041042 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:25.041197 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:25.041086 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:25.060770 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.060739 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:25.257486 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.257452 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" event={"ID":"2fb97b10-9112-4dd4-8d7d-a2c864c39358","Type":"ContainerStarted","Data":"7d9113d6c5ca2fb83888aa0f6102a53437b4750f928b28f7a5ea9826aa9c0744"} Nov 02 22:05:25.258100 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.257780 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:25.258100 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.257844 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:25.258100 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.257908 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:25.261729 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.261694 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zvx5" event={"ID":"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3","Type":"ContainerStarted","Data":"a048c61f87724f137409800738a01b51ea5d387775df0870ef099610ab2c9965"} Nov 02 22:05:25.275725 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.275690 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:25.278641 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.278614 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:25.393957 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.393923 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:05:25.394157 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.394079 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:05:25.394750 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.394727 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:05:25.523583 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.523440 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" podStartSLOduration=7.078406627 podStartE2EDuration="22.523426292s" totalImagesPullingTime="15.445019665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:25.523073275 +0000 UTC m=+23.094939163" watchObservedRunningTime="2026-11-02 22:05:25.523426292 +0000 UTC m=+23.095292177" Nov 02 22:05:25.549704 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.549665 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:25.730244 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.730213 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:25.757378 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.757331 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4zvx5" podStartSLOduration=8.342989633 podStartE2EDuration="22.757318631s" totalImagesPullingTime="14.414328998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:05:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:25.757108777 +0000 UTC m=+23.328974662" watchObservedRunningTime="2026-11-02 22:05:25.757318631 +0000 UTC m=+23.329184516" Nov 02 22:05:25.967546 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:25.967517 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:26.168057 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:26.168014 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:26.564093 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:26.564061 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:27.041113 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:27.041083 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:27.041113 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:27.041110 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:27.041384 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:27.041188 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:27.041384 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:27.041249 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:27.041384 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:27.041330 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:27.041384 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:27.041358 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:27.961819 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:27.961785 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tp2df"] Nov 02 22:05:27.962280 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:27.961916 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:27.962280 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:27.961998 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:27.966891 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:27.966841 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-92hpx"] Nov 02 22:05:27.967049 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:27.966993 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:27.967126 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:27.967103 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:28.009283 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:28.009247 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xfs82"] Nov 02 22:05:28.009453 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:28.009370 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:28.009500 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:28.009456 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:28.281260 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:28.281164 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:28.705434 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:28.705386 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:28.887955 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:28.887925 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:29.085056 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:29.084979 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:30.040493 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:30.040449 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:30.040662 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:30.040554 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:30.040662 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:30.040559 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:30.040662 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:30.040599 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:30.040760 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:30.040686 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:30.040760 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:30.040747 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:30.247267 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:30.247184 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:30.438841 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:30.438804 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:30.644294 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:30.644264 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:30.815084 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:30.815048 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:31.041517 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:31.041436 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:31.224596 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:31.224564 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:31.552122 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:31.552078 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:31.725859 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:31.725826 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:31.986084 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:31.986047 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:32.027653 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:32.027628 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vxbxf_be31cb75-89a7-4b84-b26f-2050acfa249c/dns-node-resolver/0.log" Nov 02 22:05:32.040403 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:32.040361 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:32.040490 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:32.040376 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:32.040527 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:32.040486 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:32.040591 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:32.040568 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:32.040622 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:32.040382 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:32.040677 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:32.040665 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:32.382731 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:32.382698 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:32.588307 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:32.588263 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:32.786937 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:32.786847 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:32.827764 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:32.827740 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rkr6h_8aa26f0e-38c8-4705-b64a-0a943508b307/node-ca/0.log" Nov 02 22:05:33.188123 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:33.188066 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:33.986634 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:33.986450 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:34.040491 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:34.040460 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:34.040628 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:34.040476 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:34.040628 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:34.040579 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:34.040628 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:34.040587 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:34.040724 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:34.040644 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:34.040773 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:34.040749 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:34.339148 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:34.339068 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:34.569251 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:34.569197 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:34.788916 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:34.788862 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:34.982950 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:34.982903 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:35.219509 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:35.219478 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:35.988943 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:35.988913 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:36.041169 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:36.041136 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:36.041294 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:36.041135 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:36.041294 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:36.041261 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:36.041361 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:36.041155 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:36.041361 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:36.041328 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:36.041423 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:36.041382 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:36.385032 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:36.385002 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:36.583006 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:36.582973 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:36.763440 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:36.763351 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:36.763440 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:36.763394 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:36.763637 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:36.763488 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:36.763637 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:36.763490 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:36.763637 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:36.763541 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs podName:6ae0899d-36a2-45b1-9bdf-a76acc087491 nodeName:}" failed. No retries permitted until 2026-11-02 22:06:08.763527368 +0000 UTC m=+66.335393232 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs") pod "network-metrics-daemon-xfs82" (UID: "6ae0899d-36a2-45b1-9bdf-a76acc087491") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 22:05:36.763637 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:36.763553 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret podName:644947bd-1d54-4a7f-afb6-463500cf17b9 nodeName:}" failed. No retries permitted until 2026-11-02 22:06:08.763547358 +0000 UTC m=+66.335413221 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret") pod "global-pull-secret-syncer-tp2df" (UID: "644947bd-1d54-4a7f-afb6-463500cf17b9") : object "kube-system"/"original-pull-secret" not registered Nov 02 22:05:36.782301 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:36.782265 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:36.864762 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:36.864722 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r6hz\" (UniqueName: \"kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz\") pod \"network-check-target-92hpx\" (UID: \"670c59c6-f81f-47b5-9624-2bab9d1fc847\") " pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:36.864942 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:36.864851 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 22:05:36.864942 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:36.864869 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 22:05:36.864942 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:36.864899 2581 projected.go:196] Error preparing data for projected volume kube-api-access-4r6hz for pod openshift-network-diagnostics/network-check-target-92hpx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:36.865045 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:36.864949 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz podName:670c59c6-f81f-47b5-9624-2bab9d1fc847 nodeName:}" failed. No retries permitted until 2026-11-02 22:06:08.864937545 +0000 UTC m=+66.436803409 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4r6hz" (UniqueName: "kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz") pod "network-check-target-92hpx" (UID: "670c59c6-f81f-47b5-9624-2bab9d1fc847") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 22:05:37.095784 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:37.095680 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:37.273044 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:37.273011 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:37.447385 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:37.447351 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:37.631629 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:37.631567 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:37.941022 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:37.940978 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:38.040411 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:38.040370 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:38.040411 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:38.040399 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:38.040625 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:38.040474 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:38.040625 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:38.040486 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:38.040625 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:38.040560 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:38.040730 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:38.040625 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:38.114404 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:38.114365 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:38.300069 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:38.299966 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:38.589970 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:38.589864 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:38.795440 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:38.795408 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:38.970618 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:38.970586 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:39.389315 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:39.389285 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:39.585498 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:39.585458 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:39.779708 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:39.779614 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:40.041100 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:40.041017 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:40.041100 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:40.041045 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:40.041100 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:40.041019 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:40.041384 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:40.041153 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:40.041384 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:40.041208 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:40.041384 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:40.041315 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:40.367768 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:40.367733 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:40.550823 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:40.550773 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:40.726827 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:40.726737 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:42.041059 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:42.041022 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:42.041628 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:42.041122 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:42.041628 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:42.041130 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:42.041628 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:42.041166 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:42.041628 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:42.041285 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:42.041628 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:42.041331 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:44.041110 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:44.041073 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:44.041110 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:44.041109 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:44.041555 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:44.041184 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:44.041555 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:44.041255 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:44.041555 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:44.041338 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:44.041555 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:44.041467 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:45.941885 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:45.941845 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:46.041014 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.040981 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:46.041194 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.041072 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:46.041194 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:46.041085 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92hpx" podUID="670c59c6-f81f-47b5-9624-2bab9d1fc847" Nov 02 22:05:46.041194 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.041102 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:46.041194 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:46.041188 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfs82" podUID="6ae0899d-36a2-45b1-9bdf-a76acc087491" Nov 02 22:05:46.041338 ip-10-0-140-221 kubenswrapper[2581]: E1102 22:05:46.041273 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tp2df" podUID="644947bd-1d54-4a7f-afb6-463500cf17b9" Nov 02 22:05:46.123448 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.123410 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:46.428622 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.428578 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:46.618179 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.618154 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:46.639115 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.639090 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeReady" Nov 02 22:05:46.639245 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.639209 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 22:05:46.751202 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.751169 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zqhq8"] Nov 02 22:05:46.760513 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.760488 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zqhq8" Nov 02 22:05:46.799468 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.799432 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lnqpz"] Nov 02 22:05:46.810584 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.810552 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 22:05:46.810933 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.810916 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t4sw7\"" Nov 02 22:05:46.810984 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.810918 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 22:05:46.821055 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.821024 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tnqdp"] Nov 02 22:05:46.821204 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.821165 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lnqpz" Nov 02 22:05:46.837348 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.837316 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zqhq8"] Nov 02 22:05:46.837348 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.837333 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:46.837535 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.837375 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lnqpz"] Nov 02 22:05:46.846644 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.846619 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tnqdp"] Nov 02 22:05:46.912289 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.912189 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 22:05:46.912289 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.912189 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 22:05:46.912289 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.912196 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 22:05:46.912289 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.912195 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 22:05:46.912289 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.912216 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4gtmj\"" Nov 02 22:05:46.912289 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.912260 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 22:05:46.912289 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.912267 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 22:05:46.912605 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.912336 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q5sg6\"" Nov 02 22:05:46.912605 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.912393 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 22:05:46.932054 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.932024 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfxq4\" (UniqueName: \"kubernetes.io/projected/67edaf75-8924-48b9-9a81-93089ab7c82d-kube-api-access-cfxq4\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:05:46.932121 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.932058 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67edaf75-8924-48b9-9a81-93089ab7c82d-config-volume\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:05:46.932121 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.932075 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba6a658f-a491-458f-a3be-2d71c676c069-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:46.932184 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.932153 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67edaf75-8924-48b9-9a81-93089ab7c82d-metrics-tls\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:05:46.932217 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.932187 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67edaf75-8924-48b9-9a81-93089ab7c82d-tmp-dir\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:05:46.932217 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.932211 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba6a658f-a491-458f-a3be-2d71c676c069-crio-socket\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:46.932274 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.932226 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba6a658f-a491-458f-a3be-2d71c676c069-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:46.932345 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.932328 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70d481aa-8880-41b2-957a-285385bceebd-cert\") pod \"ingress-canary-lnqpz\" (UID: \"70d481aa-8880-41b2-957a-285385bceebd\") " pod="openshift-ingress-canary/ingress-canary-lnqpz" Nov 02 22:05:46.932378 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.932366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9b2t\" (UniqueName: \"kubernetes.io/projected/ba6a658f-a491-458f-a3be-2d71c676c069-kube-api-access-w9b2t\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:46.932426 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.932414 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzqmd\" (UniqueName: \"kubernetes.io/projected/70d481aa-8880-41b2-957a-285385bceebd-kube-api-access-bzqmd\") pod \"ingress-canary-lnqpz\" (UID: \"70d481aa-8880-41b2-957a-285385bceebd\") " pod="openshift-ingress-canary/ingress-canary-lnqpz" Nov 02 22:05:46.932459 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:46.932433 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba6a658f-a491-458f-a3be-2d71c676c069-data-volume\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:47.033655 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.033618 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzqmd\" (UniqueName: \"kubernetes.io/projected/70d481aa-8880-41b2-957a-285385bceebd-kube-api-access-bzqmd\") pod \"ingress-canary-lnqpz\" (UID: \"70d481aa-8880-41b2-957a-285385bceebd\") " pod="openshift-ingress-canary/ingress-canary-lnqpz" Nov 02 22:05:47.033655 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.033653 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba6a658f-a491-458f-a3be-2d71c676c069-data-volume\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:47.034196 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.033678 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxq4\" (UniqueName: \"kubernetes.io/projected/67edaf75-8924-48b9-9a81-93089ab7c82d-kube-api-access-cfxq4\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:05:47.034196 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.033704 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67edaf75-8924-48b9-9a81-93089ab7c82d-config-volume\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:05:47.034196 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.033838 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba6a658f-a491-458f-a3be-2d71c676c069-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:47.034196 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.033951 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67edaf75-8924-48b9-9a81-93089ab7c82d-metrics-tls\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:05:47.034196 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.033990 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67edaf75-8924-48b9-9a81-93089ab7c82d-tmp-dir\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:05:47.034196 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.034036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba6a658f-a491-458f-a3be-2d71c676c069-crio-socket\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:47.034196 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.034048 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba6a658f-a491-458f-a3be-2d71c676c069-data-volume\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:47.034196 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.034067 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba6a658f-a491-458f-a3be-2d71c676c069-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:47.034196 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.034113 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70d481aa-8880-41b2-957a-285385bceebd-cert\") pod \"ingress-canary-lnqpz\" (UID: \"70d481aa-8880-41b2-957a-285385bceebd\") " pod="openshift-ingress-canary/ingress-canary-lnqpz" Nov 02 22:05:47.034196 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.034148 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9b2t\" (UniqueName: \"kubernetes.io/projected/ba6a658f-a491-458f-a3be-2d71c676c069-kube-api-access-w9b2t\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:47.034644 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.034236 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba6a658f-a491-458f-a3be-2d71c676c069-crio-socket\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:47.034644 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.034334 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67edaf75-8924-48b9-9a81-93089ab7c82d-tmp-dir\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:05:47.034644 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.034379 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67edaf75-8924-48b9-9a81-93089ab7c82d-config-volume\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:05:47.034644 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.034439 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba6a658f-a491-458f-a3be-2d71c676c069-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:47.037698 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.037669 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67edaf75-8924-48b9-9a81-93089ab7c82d-metrics-tls\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:05:47.037787 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.037674 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba6a658f-a491-458f-a3be-2d71c676c069-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:47.037862 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.037842 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70d481aa-8880-41b2-957a-285385bceebd-cert\") pod \"ingress-canary-lnqpz\" (UID: \"70d481aa-8880-41b2-957a-285385bceebd\") " pod="openshift-ingress-canary/ingress-canary-lnqpz" Nov 02 22:05:47.093108 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.093077 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfxq4\" (UniqueName: \"kubernetes.io/projected/67edaf75-8924-48b9-9a81-93089ab7c82d-kube-api-access-cfxq4\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:05:47.093272 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.093083 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9b2t\" (UniqueName: \"kubernetes.io/projected/ba6a658f-a491-458f-a3be-2d71c676c069-kube-api-access-w9b2t\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:47.093318 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.093307 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzqmd\" (UniqueName: \"kubernetes.io/projected/70d481aa-8880-41b2-957a-285385bceebd-kube-api-access-bzqmd\") pod \"ingress-canary-lnqpz\" (UID: \"70d481aa-8880-41b2-957a-285385bceebd\") " pod="openshift-ingress-canary/ingress-canary-lnqpz" Nov 02 22:05:47.144306 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.144263 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lnqpz" Nov 02 22:05:47.188370 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.188291 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:05:47.389634 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.389592 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zqhq8" Nov 02 22:05:47.523913 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.523820 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lnqpz"] Nov 02 22:05:47.531942 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:47.531909 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d481aa_8880_41b2_957a_285385bceebd.slice/crio-991641b7f1220a32684d82210fd0bf79fd19056167b58b18957d88d2973c815c WatchSource:0}: Error finding container 991641b7f1220a32684d82210fd0bf79fd19056167b58b18957d88d2973c815c: Status 404 returned error can't find the container with id 991641b7f1220a32684d82210fd0bf79fd19056167b58b18957d88d2973c815c Nov 02 22:05:47.577831 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.577799 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tnqdp"] Nov 02 22:05:47.585063 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:47.585038 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6a658f_a491_458f_a3be_2d71c676c069.slice/crio-6f2763a0e9d2236e0b2f4fe07cc15e3892b302fb52acc8d26e4bbf345c6c2ea9 WatchSource:0}: Error finding container 6f2763a0e9d2236e0b2f4fe07cc15e3892b302fb52acc8d26e4bbf345c6c2ea9: Status 404 returned error can't find the container with id 6f2763a0e9d2236e0b2f4fe07cc15e3892b302fb52acc8d26e4bbf345c6c2ea9 Nov 02 22:05:47.755750 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:47.755714 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zqhq8"] Nov 02 22:05:47.763265 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:47.763237 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67edaf75_8924_48b9_9a81_93089ab7c82d.slice/crio-052c047e99f57ddbabb7d762bf82fd0f0574338e430f070867180cc7f2dcb417 WatchSource:0}: Error finding container 052c047e99f57ddbabb7d762bf82fd0f0574338e430f070867180cc7f2dcb417: Status 404 returned error can't find the container with id 052c047e99f57ddbabb7d762bf82fd0f0574338e430f070867180cc7f2dcb417 Nov 02 22:05:48.041030 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:48.040990 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:05:48.041460 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:48.041106 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:05:48.041460 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:48.041137 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:05:48.091719 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:48.091687 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:05:48.091906 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:48.091861 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l4knj\"" Nov 02 22:05:48.091906 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:48.091896 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:05:48.092048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:48.092002 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6jxx8\"" Nov 02 22:05:48.092048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:48.092007 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g764w\"" Nov 02 22:05:48.092125 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:48.092055 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:05:48.092125 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:48.092099 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:05:48.315726 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:48.315633 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zqhq8" event={"ID":"67edaf75-8924-48b9-9a81-93089ab7c82d","Type":"ContainerStarted","Data":"052c047e99f57ddbabb7d762bf82fd0f0574338e430f070867180cc7f2dcb417"} Nov 02 22:05:48.316729 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:48.316703 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lnqpz" event={"ID":"70d481aa-8880-41b2-957a-285385bceebd","Type":"ContainerStarted","Data":"991641b7f1220a32684d82210fd0bf79fd19056167b58b18957d88d2973c815c"} Nov 02 22:05:48.318034 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:48.318015 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tnqdp" event={"ID":"ba6a658f-a491-458f-a3be-2d71c676c069","Type":"ContainerStarted","Data":"ebdabb67d7d57f99e4fb54c37220df38f2d3bafe10ba83fa840bcc956bad92d9"} Nov 02 22:05:48.318116 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:48.318052 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tnqdp" event={"ID":"ba6a658f-a491-458f-a3be-2d71c676c069","Type":"ContainerStarted","Data":"6f2763a0e9d2236e0b2f4fe07cc15e3892b302fb52acc8d26e4bbf345c6c2ea9"} Nov 02 22:05:49.322911 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:49.322814 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tnqdp" event={"ID":"ba6a658f-a491-458f-a3be-2d71c676c069","Type":"ContainerStarted","Data":"449c0953337f0af260979d99542915bfd69c1096b07751ae3a83ae4c67267bea"} Nov 02 22:05:49.563511 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:49.563438 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:49.739415 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:49.739370 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:49.910698 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:49.910665 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:50.084688 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:50.084611 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:50.330898 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:50.328295 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zqhq8" event={"ID":"67edaf75-8924-48b9-9a81-93089ab7c82d","Type":"ContainerStarted","Data":"fde8825ac9814e2869dd2e5b1ea9d574d03ea2a81270c9ff9e038dac0324f65e"} Nov 02 22:05:51.332383 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.332289 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zqhq8" event={"ID":"67edaf75-8924-48b9-9a81-93089ab7c82d","Type":"ContainerStarted","Data":"bfa9dfd6cef3c99a879cf032787b89cc92a901b51cf0612ec2b4e5cbc22e79ea"} Nov 02 22:05:51.332891 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.332403 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zqhq8" Nov 02 22:05:51.333978 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.333947 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lnqpz" event={"ID":"70d481aa-8880-41b2-957a-285385bceebd","Type":"ContainerStarted","Data":"d51c75dd485f1ba674b5ae9088adc77b38e8a79d7ecd937eb294ac72138a4ea9"} Nov 02 22:05:51.335865 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.335842 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tnqdp" event={"ID":"ba6a658f-a491-458f-a3be-2d71c676c069","Type":"ContainerStarted","Data":"628166c6c0865851ef93cf5dc99fee45ec93355926f9e0f1459b28bc637169b2"} Nov 02 22:05:51.461785 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.461727 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-zqhq8" podStartSLOduration=3.019078694 podStartE2EDuration="5.461713705s" totalImagesPullingTime="2.442635011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:51.461168305 +0000 UTC m=+49.033034186" watchObservedRunningTime="2026-11-02 22:05:51.461713705 +0000 UTC m=+49.033579638" Nov 02 22:05:51.572612 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.572556 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lnqpz" podStartSLOduration=2.034809524 podStartE2EDuration="5.572541774s" totalImagesPullingTime="3.53773225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:51.571940382 +0000 UTC m=+49.143806267" watchObservedRunningTime="2026-11-02 22:05:51.572541774 +0000 UTC m=+49.144407654" Nov 02 22:05:51.687064 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.687010 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tnqdp" podStartSLOduration=2.379073699 podStartE2EDuration="5.686993397s" totalImagesPullingTime="3.307919698s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:05:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:51.686331812 +0000 UTC m=+49.258197698" watchObservedRunningTime="2026-11-02 22:05:51.686993397 +0000 UTC m=+49.258859282" Nov 02 22:05:51.814630 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.814594 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6967dc6984-c2sqx"] Nov 02 22:05:51.816913 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.816894 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:51.867582 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.867546 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 22:05:51.867582 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.867546 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 22:05:51.867858 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.867640 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9shnq\"" Nov 02 22:05:51.867858 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.867767 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 22:05:51.867858 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.867794 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 22:05:51.867858 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.867803 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 22:05:51.867858 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.867817 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 22:05:51.867858 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.867848 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 22:05:51.874018 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.873989 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6967dc6984-c2sqx"] Nov 02 22:05:51.974073 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.973977 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-oauth-serving-cert\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:51.974073 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.974028 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-oauth-config\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:51.974259 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.974119 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-service-ca\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:51.974259 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.974143 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-serving-cert\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:51.974259 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.974164 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-config\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:51.974348 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.974257 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-tmp\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:51.974348 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:51.974305 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nbpz\" (UniqueName: \"kubernetes.io/projected/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-kube-api-access-8nbpz\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:52.074645 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:52.074603 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nbpz\" (UniqueName: \"kubernetes.io/projected/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-kube-api-access-8nbpz\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:52.074645 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:52.074647 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-oauth-serving-cert\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:52.074860 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:52.074690 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-oauth-config\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:52.074860 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:52.074718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-service-ca\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:52.074860 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:52.074735 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-serving-cert\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:52.074860 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:52.074750 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-config\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:52.074860 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:52.074780 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-tmp\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:52.075317 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:52.075286 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-tmp\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:52.075560 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:52.075541 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-config\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:52.075696 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:52.075674 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-service-ca\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:52.075762 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:52.075681 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-oauth-serving-cert\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:52.077275 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:52.077253 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-oauth-config\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:52.077373 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:52.077329 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-serving-cert\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:52.133969 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:52.133933 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nbpz\" (UniqueName: \"kubernetes.io/projected/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-kube-api-access-8nbpz\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:52.427380 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:52.427327 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:05:52.795991 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:52.795893 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6967dc6984-c2sqx"] Nov 02 22:05:52.834004 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:52.833976 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3102b4_bfe2_49a1_aa23_1c7b451f0da1.slice/crio-46409b905adf9a74d7a4145f7092b909ace522c82b374060c92da1474e1ce7f8 WatchSource:0}: Error finding container 46409b905adf9a74d7a4145f7092b909ace522c82b374060c92da1474e1ce7f8: Status 404 returned error can't find the container with id 46409b905adf9a74d7a4145f7092b909ace522c82b374060c92da1474e1ce7f8 Nov 02 22:05:53.341724 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:53.341690 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6967dc6984-c2sqx" event={"ID":"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1","Type":"ContainerStarted","Data":"46409b905adf9a74d7a4145f7092b909ace522c82b374060c92da1474e1ce7f8"} Nov 02 22:05:54.355457 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.355420 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:54.449766 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.449729 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-59b677bf6c-t8dfh"] Nov 02 22:05:54.454092 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.453223 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.534609 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.534554 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:54.593016 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.592979 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-trusted-ca-bundle\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.593016 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.593021 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-config\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.593323 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.593113 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbnh2\" (UniqueName: \"kubernetes.io/projected/e95934d1-f5df-4c06-ab39-5e7c8d58858b-kube-api-access-gbnh2\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.593323 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.593174 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-serving-cert\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.593323 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.593204 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e95934d1-f5df-4c06-ab39-5e7c8d58858b-tmp\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.593323 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.593299 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-service-ca\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.593445 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.593350 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-oauth-serving-cert\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.593445 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.593384 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-oauth-config\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.648075 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.648037 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 22:05:54.648925 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.648901 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59b677bf6c-t8dfh"] Nov 02 22:05:54.694279 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.694233 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbnh2\" (UniqueName: \"kubernetes.io/projected/e95934d1-f5df-4c06-ab39-5e7c8d58858b-kube-api-access-gbnh2\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.694279 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.694283 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-serving-cert\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.694494 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.694311 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e95934d1-f5df-4c06-ab39-5e7c8d58858b-tmp\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.694494 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.694346 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-service-ca\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.694494 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.694377 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-oauth-serving-cert\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.694494 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.694401 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-oauth-config\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.694494 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.694426 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-trusted-ca-bundle\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.694494 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.694442 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-config\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.695275 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.695235 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-config\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.695625 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.695576 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e95934d1-f5df-4c06-ab39-5e7c8d58858b-tmp\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.695718 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.695650 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-service-ca\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.695775 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.695716 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-oauth-serving-cert\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.696523 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.696496 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-trusted-ca-bundle\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.700413 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.700384 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-oauth-config\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.702405 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.702380 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-serving-cert\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.756203 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.756170 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbnh2\" (UniqueName: \"kubernetes.io/projected/e95934d1-f5df-4c06-ab39-5e7c8d58858b-kube-api-access-gbnh2\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:54.765968 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:54.765938 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:05:55.144556 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:55.144517 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59b677bf6c-t8dfh"] Nov 02 22:05:55.152995 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:55.152965 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95934d1_f5df_4c06_ab39_5e7c8d58858b.slice/crio-25e3292d0c67e6de4f616aafcfb0b0051f79b82ca336b9839f160f9a4298a8e6 WatchSource:0}: Error finding container 25e3292d0c67e6de4f616aafcfb0b0051f79b82ca336b9839f160f9a4298a8e6: Status 404 returned error can't find the container with id 25e3292d0c67e6de4f616aafcfb0b0051f79b82ca336b9839f160f9a4298a8e6 Nov 02 22:05:55.348981 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:55.348944 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b677bf6c-t8dfh" event={"ID":"e95934d1-f5df-4c06-ab39-5e7c8d58858b","Type":"ContainerStarted","Data":"25e3292d0c67e6de4f616aafcfb0b0051f79b82ca336b9839f160f9a4298a8e6"} Nov 02 22:05:55.559820 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:55.559732 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:55.735084 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:55.735045 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:55.909987 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:55.909949 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:56.355421 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:56.355343 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6967dc6984-c2sqx" event={"ID":"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1","Type":"ContainerStarted","Data":"a2d1e014f1e3fc12e80006fda5de0de12b000056d6fd15a8742f1ba2d7f6c7b9"} Nov 02 22:05:56.469664 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:56.469607 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6967dc6984-c2sqx" podStartSLOduration=2.230693189 podStartE2EDuration="5.469577555s" totalImagesPullingTime="3.238884366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:56.469112177 +0000 UTC m=+54.040978063" watchObservedRunningTime="2026-11-02 22:05:56.469577555 +0000 UTC m=+54.041443440" Nov 02 22:05:56.913456 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:56.913421 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:57.120630 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.120575 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:57.277317 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.277240 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:05:57.297916 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.297882 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:57.363576 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.363531 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b677bf6c-t8dfh" event={"ID":"e95934d1-f5df-4c06-ab39-5e7c8d58858b","Type":"ContainerStarted","Data":"05ec70b6305730f6fbd0e9af1b2a15697a47251e2109fb1a219e7873db2c4c49"} Nov 02 22:05:57.487039 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.487005 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:57.546299 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.546209 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs"] Nov 02 22:05:57.548664 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.548647 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:05:57.577140 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.577100 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs"] Nov 02 22:05:57.602335 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.602297 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 22:05:57.602335 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.602314 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 22:05:57.602526 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.602388 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 22:05:57.602853 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.602837 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 22:05:57.602961 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.602947 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 22:05:57.603024 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.603008 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cmpr7\"" Nov 02 22:05:57.610152 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.610123 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w"] Nov 02 22:05:57.612703 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.612682 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.613080 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.613028 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-59b677bf6c-t8dfh" podStartSLOduration=1.48153227 podStartE2EDuration="3.613013842s" totalImagesPullingTime="2.131481572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:05:57.612627291 +0000 UTC m=+55.184493178" watchObservedRunningTime="2026-11-02 22:05:57.613013842 +0000 UTC m=+55.184879730" Nov 02 22:05:57.623459 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.623434 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gg8nx"] Nov 02 22:05:57.625641 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.625622 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.671065 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.671031 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:57.681305 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.681276 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hz4mv\"" Nov 02 22:05:57.681453 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.681410 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 22:05:57.681453 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.681441 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 22:05:57.681552 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.681444 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 22:05:57.681552 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.681488 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v5dnx\"" Nov 02 22:05:57.681552 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.681495 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 22:05:57.683891 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.683847 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w"] Nov 02 22:05:57.701689 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.701655 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 22:05:57.701845 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.701801 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 22:05:57.723423 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.723386 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.723603 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.723547 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec8310e5-d852-44c1-901b-b54a6c13a798-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:05:57.723687 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.723625 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-sys\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.723687 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.723665 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-wtmp\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.723800 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.723773 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gwv8\" (UniqueName: \"kubernetes.io/projected/ec8310e5-d852-44c1-901b-b54a6c13a798-kube-api-access-5gwv8\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:05:57.723840 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.723824 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.723954 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.723938 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xbbh\" (UniqueName: \"kubernetes.io/projected/759f8e6e-92ee-4b83-9447-949cb34a2cd1-kube-api-access-4xbbh\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.723999 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.723982 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec8310e5-d852-44c1-901b-b54a6c13a798-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:05:57.724062 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.724040 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec8310e5-d852-44c1-901b-b54a6c13a798-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:05:57.724098 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.724088 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.724137 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.724114 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2ph7\" (UniqueName: \"kubernetes.io/projected/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-api-access-c2ph7\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.724175 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.724145 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-root\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.724209 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.724183 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-textfile\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.724247 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.724214 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.724277 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.724260 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-accelerators-collector-config\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.724312 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.724284 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.724348 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.724331 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.724387 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.724358 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/759f8e6e-92ee-4b83-9447-949cb34a2cd1-metrics-client-ca\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.724420 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.724389 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-tls\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.825212 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825112 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-accelerators-collector-config\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.825212 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.825212 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825181 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.825212 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825208 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/759f8e6e-92ee-4b83-9447-949cb34a2cd1-metrics-client-ca\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.825521 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825227 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-tls\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.825521 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825264 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.825521 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825315 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec8310e5-d852-44c1-901b-b54a6c13a798-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:05:57.825521 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825351 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-sys\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.825521 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825390 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-wtmp\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.825521 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825435 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwv8\" (UniqueName: \"kubernetes.io/projected/ec8310e5-d852-44c1-901b-b54a6c13a798-kube-api-access-5gwv8\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:05:57.825521 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825446 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-sys\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.825521 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825503 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.825943 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825547 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xbbh\" (UniqueName: \"kubernetes.io/projected/759f8e6e-92ee-4b83-9447-949cb34a2cd1-kube-api-access-4xbbh\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.825943 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec8310e5-d852-44c1-901b-b54a6c13a798-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:05:57.825943 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825637 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec8310e5-d852-44c1-901b-b54a6c13a798-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:05:57.825943 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825660 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.825943 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825684 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2ph7\" (UniqueName: \"kubernetes.io/projected/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-api-access-c2ph7\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.825943 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825712 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-root\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.825943 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825750 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-textfile\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.825943 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825785 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.825943 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.825868 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-wtmp\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.826381 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.826100 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.826640 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.826611 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec8310e5-d852-44c1-901b-b54a6c13a798-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:05:57.827359 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.826690 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-accelerators-collector-config\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.827456 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.826710 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-root\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.827606 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.826976 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.827707 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.827112 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-textfile\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.827707 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.827150 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/759f8e6e-92ee-4b83-9447-949cb34a2cd1-metrics-client-ca\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.827909 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.827348 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.828305 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.828280 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.828479 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.828456 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-tls\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.828545 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.828492 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec8310e5-d852-44c1-901b-b54a6c13a798-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:05:57.828545 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.828519 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.829424 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.829402 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec8310e5-d852-44c1-901b-b54a6c13a798-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:05:57.829502 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.829422 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.849868 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.849807 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:57.885409 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.885375 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gwv8\" (UniqueName: \"kubernetes.io/projected/ec8310e5-d852-44c1-901b-b54a6c13a798-kube-api-access-5gwv8\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:05:57.886646 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.886607 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2ph7\" (UniqueName: \"kubernetes.io/projected/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-api-access-c2ph7\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.886775 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.886664 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xbbh\" (UniqueName: \"kubernetes.io/projected/759f8e6e-92ee-4b83-9447-949cb34a2cd1-kube-api-access-4xbbh\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.926596 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.926550 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:05:57.938269 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:57.938240 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:05:57.947510 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:57.947451 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759f8e6e_92ee_4b83_9447_949cb34a2cd1.slice/crio-501126d919c6c9ab589aecebf1bbab791fb4ff4249ddde3f7ee12e171df7c5f6 WatchSource:0}: Error finding container 501126d919c6c9ab589aecebf1bbab791fb4ff4249ddde3f7ee12e171df7c5f6: Status 404 returned error can't find the container with id 501126d919c6c9ab589aecebf1bbab791fb4ff4249ddde3f7ee12e171df7c5f6 Nov 02 22:05:58.027965 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:58.027930 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:58.161500 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:58.161463 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:05:58.216166 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:58.216140 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:58.301820 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:58.301791 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w"] Nov 02 22:05:58.309109 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:58.309074 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3ab9d6_1afa_499c_80d2_a1d11981eeb2.slice/crio-363b3db111d77b5a6fc5447bdc646228edf2e548ef89f06617c4ade0a6a00c4a WatchSource:0}: Error finding container 363b3db111d77b5a6fc5447bdc646228edf2e548ef89f06617c4ade0a6a00c4a: Status 404 returned error can't find the container with id 363b3db111d77b5a6fc5447bdc646228edf2e548ef89f06617c4ade0a6a00c4a Nov 02 22:05:58.368360 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:58.368325 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" event={"ID":"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2","Type":"ContainerStarted","Data":"363b3db111d77b5a6fc5447bdc646228edf2e548ef89f06617c4ade0a6a00c4a"} Nov 02 22:05:58.369572 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:58.369547 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gg8nx" event={"ID":"759f8e6e-92ee-4b83-9447-949cb34a2cd1","Type":"ContainerStarted","Data":"501126d919c6c9ab589aecebf1bbab791fb4ff4249ddde3f7ee12e171df7c5f6"} Nov 02 22:05:58.402105 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:58.402073 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:58.542367 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:58.541135 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs"] Nov 02 22:05:58.547289 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:05:58.547260 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8310e5_d852_44c1_901b_b54a6c13a798.slice/crio-a5c2101529a22c04fa525c591d93dc1f5982dfcf7a252b9603227560914c4208 WatchSource:0}: Error finding container a5c2101529a22c04fa525c591d93dc1f5982dfcf7a252b9603227560914c4208: Status 404 returned error can't find the container with id a5c2101529a22c04fa525c591d93dc1f5982dfcf7a252b9603227560914c4208 Nov 02 22:05:58.994347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:58.994310 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:59.219240 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:59.219203 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:59.375890 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:59.375853 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" event={"ID":"ec8310e5-d852-44c1-901b-b54a6c13a798","Type":"ContainerStarted","Data":"9d0b505e556cbfcf06a0cd813c2ab50d465764e356c54b2f847e43e8181bade2"} Nov 02 22:05:59.376062 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:59.375942 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" event={"ID":"ec8310e5-d852-44c1-901b-b54a6c13a798","Type":"ContainerStarted","Data":"6d1fb7fdaa9744bc8ddfef156adda9844b213629744df8370d6dae143c8a1e8b"} Nov 02 22:05:59.376062 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:59.375972 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" event={"ID":"ec8310e5-d852-44c1-901b-b54a6c13a798","Type":"ContainerStarted","Data":"a5c2101529a22c04fa525c591d93dc1f5982dfcf7a252b9603227560914c4208"} Nov 02 22:05:59.869307 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:59.869273 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:05:59.889596 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:59.889566 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:05:59.892574 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:59.892547 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:05:59.943684 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:59.943622 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 22:05:59.943684 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:59.943622 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 22:05:59.943684 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:59.943652 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 22:05:59.943684 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:59.943668 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 22:05:59.944110 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:59.943698 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 22:05:59.991234 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:59.991124 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 22:05:59.991234 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:05:59.991132 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 22:06:00.039759 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.039725 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 22:06:00.041269 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.041244 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:06:00.043751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.043726 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 22:06:00.043929 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.043902 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6jjrn\"" Nov 02 22:06:00.044036 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.044010 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.044093 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.044069 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.044144 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.044108 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sql9\" (UniqueName: \"kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-kube-api-access-4sql9\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.044144 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.044125 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.044144 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.044142 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-out\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.044346 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.044179 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.044346 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.044197 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.044346 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.044230 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.044346 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.044259 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.044346 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.044275 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.044346 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.044318 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.044631 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.044354 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-web-config\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.044631 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.044376 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.145728 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.145685 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.145957 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.145736 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-web-config\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.145957 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.145763 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.145957 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.145806 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.145957 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.145842 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.145957 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.145912 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sql9\" (UniqueName: \"kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-kube-api-access-4sql9\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.145957 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.145936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.146272 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.145961 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-out\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.146272 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.146006 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.146272 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.146036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.146272 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.146095 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.146272 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.146129 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.146272 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.146152 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.146817 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.146790 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.148522 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.148484 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.148960 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.148917 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.151304 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.151222 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.152493 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.152446 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.153179 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.153115 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.153179 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.153123 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-web-config\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.153788 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.153763 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.154483 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.154456 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.155520 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.155482 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.155798 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.155779 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.162346 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.162318 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-out\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.207956 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.207921 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sql9\" (UniqueName: \"kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-kube-api-access-4sql9\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.216510 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.216478 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:00.385544 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.385505 2581 generic.go:356] "Generic (PLEG): container finished" podID="759f8e6e-92ee-4b83-9447-949cb34a2cd1" containerID="769540672d397b834678476587d94a7646470538b21f4acc7687782638914785" exitCode=0 Nov 02 22:06:00.385922 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.385723 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gg8nx" event={"ID":"759f8e6e-92ee-4b83-9447-949cb34a2cd1","Type":"ContainerDied","Data":"769540672d397b834678476587d94a7646470538b21f4acc7687782638914785"} Nov 02 22:06:00.388708 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.388678 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" event={"ID":"ec8310e5-d852-44c1-901b-b54a6c13a798","Type":"ContainerStarted","Data":"614b43e0e611f7117deb0895f6c7e80de75df33557e36a7931129d2fc1b7c21d"} Nov 02 22:06:00.391589 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.391504 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" event={"ID":"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2","Type":"ContainerStarted","Data":"236398f99e70c4b1face261d133548a6ec0dc3fe3641efca106b52af447865ff"} Nov 02 22:06:00.391589 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.391563 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" event={"ID":"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2","Type":"ContainerStarted","Data":"7f3a57f8efe483152c6ba4c0a315045a579d399f5044d6cb7ce30712668d09f8"} Nov 02 22:06:00.562980 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.562949 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:00.614773 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.614720 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" podStartSLOduration=1.982213007 podStartE2EDuration="3.614706031s" totalImagesPullingTime="1.632493024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:00.614476999 +0000 UTC m=+58.186342885" watchObservedRunningTime="2026-11-02 22:06:00.614706031 +0000 UTC m=+58.186571933" Nov 02 22:06:00.679665 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.679588 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:06:00.687261 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:06:00.687229 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d7bd65_c3af_49fa_9c77_8c7c348e73f4.slice/crio-688a334dbdb09643b4eeffbd2800ec19eb51db27b196952cc2361e0226cd28e7 WatchSource:0}: Error finding container 688a334dbdb09643b4eeffbd2800ec19eb51db27b196952cc2361e0226cd28e7: Status 404 returned error can't find the container with id 688a334dbdb09643b4eeffbd2800ec19eb51db27b196952cc2361e0226cd28e7 Nov 02 22:06:00.763586 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:00.763551 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:01.340842 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:01.340809 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zqhq8" Nov 02 22:06:01.397075 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:01.397039 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" event={"ID":"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2","Type":"ContainerStarted","Data":"5a550c10014f8037107289e00b030ff25cab1d0eb3e799ca4bc65a39fa0aeefb"} Nov 02 22:06:01.398274 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:01.398244 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4","Type":"ContainerStarted","Data":"688a334dbdb09643b4eeffbd2800ec19eb51db27b196952cc2361e0226cd28e7"} Nov 02 22:06:01.400276 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:01.400246 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gg8nx" event={"ID":"759f8e6e-92ee-4b83-9447-949cb34a2cd1","Type":"ContainerStarted","Data":"8ef36603057dbe4f375dd21d170b39a54ffb0ea4e70acc1019b68db8a8a4b1c4"} Nov 02 22:06:01.400405 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:01.400303 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gg8nx" event={"ID":"759f8e6e-92ee-4b83-9447-949cb34a2cd1","Type":"ContainerStarted","Data":"5eea9d5cfa1673bf445d45177cf4914a541a41d5b39bc95c9504fca2fefc94f1"} Nov 02 22:06:01.570621 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:01.570565 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" podStartSLOduration=2.668528422 podStartE2EDuration="4.570550073s" totalImagesPullingTime="1.902021651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:01.569910221 +0000 UTC m=+59.141776107" watchObservedRunningTime="2026-11-02 22:06:01.570550073 +0000 UTC m=+59.142415961" Nov 02 22:06:01.685756 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:01.685693 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gg8nx" podStartSLOduration=2.901153105 podStartE2EDuration="4.685679614s" totalImagesPullingTime="1.699906168s" totalInitContainerRuntime="84.620341ms" isStatefulPod=true podCreationTimestamp="2026-11-02 22:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:01.684292385 +0000 UTC m=+59.256158271" watchObservedRunningTime="2026-11-02 22:06:01.685679614 +0000 UTC m=+59.257545497" Nov 02 22:06:01.727376 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:01.727321 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:01.906460 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:01.906426 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:02.111576 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.111442 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:02.288653 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.288620 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:02.427752 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.427724 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:02.428113 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.427800 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:02.433693 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.433669 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:02.465139 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.465113 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:02.515351 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.515318 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7bff846db6-xs49b"] Nov 02 22:06:02.517645 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.517625 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.568217 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.568170 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b199a04-e491-43c9-baf3-0d9e6c663dfa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.568376 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.568224 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 22:06:02.568376 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.568240 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xhjpw\"" Nov 02 22:06:02.568376 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.568282 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-client-ca-bundle\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.568376 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.568315 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7b199a04-e491-43c9-baf3-0d9e6c663dfa-audit-log\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.568376 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.568364 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-secret-metrics-server-tls\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.568574 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.568413 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-secret-metrics-server-client-certs\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.568574 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.568443 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7b199a04-e491-43c9-baf3-0d9e6c663dfa-metrics-server-audit-profiles\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.568574 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.568546 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjx2f\" (UniqueName: \"kubernetes.io/projected/7b199a04-e491-43c9-baf3-0d9e6c663dfa-kube-api-access-vjx2f\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.574487 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.574456 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bff846db6-xs49b"] Nov 02 22:06:02.586202 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.586161 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 22:06:02.586360 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.586233 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 22:06:02.645455 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.645368 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:02.668713 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.668675 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 22:06:02.668904 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.668795 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-88l269il0r23v\"" Nov 02 22:06:02.668958 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.668928 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b199a04-e491-43c9-baf3-0d9e6c663dfa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.668999 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.668984 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-client-ca-bundle\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.669048 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.669016 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7b199a04-e491-43c9-baf3-0d9e6c663dfa-audit-log\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.669091 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.669077 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-secret-metrics-server-tls\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.669152 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.669130 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-secret-metrics-server-client-certs\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.669275 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.669209 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7b199a04-e491-43c9-baf3-0d9e6c663dfa-metrics-server-audit-profiles\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.669275 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.669267 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjx2f\" (UniqueName: \"kubernetes.io/projected/7b199a04-e491-43c9-baf3-0d9e6c663dfa-kube-api-access-vjx2f\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.669555 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.669533 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7b199a04-e491-43c9-baf3-0d9e6c663dfa-audit-log\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.669831 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.669807 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b199a04-e491-43c9-baf3-0d9e6c663dfa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.670077 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.670057 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7b199a04-e491-43c9-baf3-0d9e6c663dfa-metrics-server-audit-profiles\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.671667 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.671647 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-secret-metrics-server-client-certs\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.671783 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.671767 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-client-ca-bundle\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.671820 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.671800 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-secret-metrics-server-tls\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.728372 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.728337 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjx2f\" (UniqueName: \"kubernetes.io/projected/7b199a04-e491-43c9-baf3-0d9e6c663dfa-kube-api-access-vjx2f\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:02.842330 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:02.842275 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:03.263461 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:03.263377 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bff846db6-xs49b"] Nov 02 22:06:03.270420 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:06:03.270382 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b199a04_e491_43c9_baf3_0d9e6c663dfa.slice/crio-16b570a4eb4094d0ad6ec1911e23a79c1fd097ff8f379536f786416721565d99 WatchSource:0}: Error finding container 16b570a4eb4094d0ad6ec1911e23a79c1fd097ff8f379536f786416721565d99: Status 404 returned error can't find the container with id 16b570a4eb4094d0ad6ec1911e23a79c1fd097ff8f379536f786416721565d99 Nov 02 22:06:03.408014 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:03.407976 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" event={"ID":"7b199a04-e491-43c9-baf3-0d9e6c663dfa","Type":"ContainerStarted","Data":"16b570a4eb4094d0ad6ec1911e23a79c1fd097ff8f379536f786416721565d99"} Nov 02 22:06:03.409552 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:03.409526 2581 generic.go:356] "Generic (PLEG): container finished" podID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerID="5064217c1727c131b0e57008af6162977bb22e38ebaf8db195cb216cf2f4ee4d" exitCode=0 Nov 02 22:06:03.409673 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:03.409614 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4","Type":"ContainerDied","Data":"5064217c1727c131b0e57008af6162977bb22e38ebaf8db195cb216cf2f4ee4d"} Nov 02 22:06:03.413751 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:03.413730 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:03.592236 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:03.592157 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:03.770136 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:03.770100 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:04.383828 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:04.383791 2581 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59b677bf6c-t8dfh"] Nov 02 22:06:04.766539 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:04.766431 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:06.424149 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.424111 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4","Type":"ContainerStarted","Data":"dc00cab91a4160629f3a6a7df16ffbfa5feaa5d34c1f42c27d0a9f62996b5207"} Nov 02 22:06:06.424559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.424199 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4","Type":"ContainerStarted","Data":"b0c2b65626f6d3796317124b61409892a23efe75bb852073d089f2314e02a792"} Nov 02 22:06:06.424559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.424240 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4","Type":"ContainerStarted","Data":"4c6bfcbe1ed2682d3f93aa1b9f6cebed5f6034ac176331a328f14b458fe56582"} Nov 02 22:06:06.424559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.424282 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4","Type":"ContainerStarted","Data":"5891915245a856c716b3457d18b77ebc7a28274b31169b43ce4a646d3728ef74"} Nov 02 22:06:06.424559 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.424324 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4","Type":"ContainerStarted","Data":"b26d63ec309e5fe09be31f4a05a4bea233fd45eca992aa8e51fd2d6e209ed189"} Nov 02 22:06:06.425719 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.425694 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" event={"ID":"7b199a04-e491-43c9-baf3-0d9e6c663dfa","Type":"ContainerStarted","Data":"b711a0f8776026f901b1d8e2496b5d6ab7a3088100de481d66b32bc7db57a5d8"} Nov 02 22:06:06.539628 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.539576 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" podStartSLOduration=2.32105145 podStartE2EDuration="4.539561528s" totalImagesPullingTime="2.218510078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:06.538986829 +0000 UTC m=+64.110852717" watchObservedRunningTime="2026-11-02 22:06:06.539561528 +0000 UTC m=+64.111427411" Nov 02 22:06:06.693728 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.693649 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:06.694027 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.694006 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:06:06.700235 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.700197 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.751150 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.751071 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 22:06:06.751150 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.751100 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 22:06:06.751150 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.751127 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 22:06:06.805890 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.805828 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.805898 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.805986 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806014 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806033 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806063 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806051 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806235 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806146 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-config\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806235 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806170 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-config-out\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806235 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806187 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806369 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806277 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806369 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806336 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-web-config\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806464 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806367 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806464 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806434 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806553 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806466 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806553 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806505 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806553 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806533 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806711 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806573 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806711 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806596 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806711 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806660 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbw9d\" (UniqueName: \"kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-kube-api-access-dbw9d\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.806711 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.806695 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.846833 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.846779 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 22:06:06.847413 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.847390 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 22:06:06.847536 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.847521 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 22:06:06.847615 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.847541 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 22:06:06.847615 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.847578 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dmih5pep9rcoc\"" Nov 02 22:06:06.847615 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.847573 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 22:06:06.847615 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.847602 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 22:06:06.847757 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.847664 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 22:06:06.848977 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.848959 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:06:06.888167 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.888130 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 22:06:06.888167 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.888130 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 22:06:06.890594 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.890573 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 22:06:06.890594 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.890585 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vgqck\"" Nov 02 22:06:06.890735 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.890599 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 22:06:06.907942 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.907906 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbw9d\" (UniqueName: \"kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-kube-api-access-dbw9d\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908093 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.907951 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908169 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908108 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908169 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908152 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908273 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908193 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908273 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908208 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908273 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908226 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908273 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908254 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908451 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908295 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-config\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908451 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908318 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-config-out\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908451 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908340 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908451 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908366 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908451 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908392 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-web-config\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908451 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908414 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908726 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908451 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908726 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908484 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908726 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908523 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908726 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908549 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908726 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908584 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.908726 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908619 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.909120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908860 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.909120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908970 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.909120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.908993 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.909120 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.909023 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.909339 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.909245 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.910363 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.910328 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.911581 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.910496 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.912341 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.911842 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-config\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.912341 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.912297 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.912832 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.912803 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.912963 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.912814 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.912963 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.912936 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-web-config\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.913407 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.913309 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.913613 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.913587 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.913829 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.913810 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-config-out\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.913919 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.913828 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.914103 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.914085 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.914424 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.914407 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.914812 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.914794 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:06.967406 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:06.967331 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbw9d\" (UniqueName: \"kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-kube-api-access-dbw9d\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:07.020174 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:07.020136 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:07.458914 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:07.458867 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:06:07.465539 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:06:07.465507 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272c657c_1cb5_4f20_a3b7_035c535c2498.slice/crio-be03c1c4bf106b4e190ff5861d9f12fb41fd9db1b7bbca2dc2ab18a2bee7bdca WatchSource:0}: Error finding container be03c1c4bf106b4e190ff5861d9f12fb41fd9db1b7bbca2dc2ab18a2bee7bdca: Status 404 returned error can't find the container with id be03c1c4bf106b4e190ff5861d9f12fb41fd9db1b7bbca2dc2ab18a2bee7bdca Nov 02 22:06:08.399072 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:08.399034 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:08.436708 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:08.436670 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4","Type":"ContainerStarted","Data":"2da91375ef30cbd47ccd84f5d23c10081042db722eb215c8af62441e3e2f6684"} Nov 02 22:06:08.438399 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:08.438371 2581 generic.go:356] "Generic (PLEG): container finished" podID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerID="368273d4a42d7eae3ebb350aa7b1a269bcedd1b8a57eefb77aa54f03d4f5d6d1" exitCode=0 Nov 02 22:06:08.438547 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:08.438434 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"272c657c-1cb5-4f20-a3b7-035c535c2498","Type":"ContainerDied","Data":"368273d4a42d7eae3ebb350aa7b1a269bcedd1b8a57eefb77aa54f03d4f5d6d1"} Nov 02 22:06:08.438547 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:08.438485 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"272c657c-1cb5-4f20-a3b7-035c535c2498","Type":"ContainerStarted","Data":"be03c1c4bf106b4e190ff5861d9f12fb41fd9db1b7bbca2dc2ab18a2bee7bdca"} Nov 02 22:06:08.568068 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:08.568002 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.729231945 podStartE2EDuration="9.567983742s" totalImagesPullingTime="5.823283516s" totalInitContainerRuntime="15.468281ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:08.561848634 +0000 UTC m=+66.133714520" watchObservedRunningTime="2026-11-02 22:06:08.567983742 +0000 UTC m=+66.139849628" Nov 02 22:06:08.582309 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:08.582263 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:08.793860 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:08.793781 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:08.827290 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:08.827251 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:06:08.827439 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:08.827349 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:06:08.876592 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:08.876559 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:06:08.876592 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:08.876559 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:06:08.879958 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:08.879931 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:06:08.879958 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:08.879948 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:06:08.928397 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:08.928358 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r6hz\" (UniqueName: \"kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz\") pod \"network-check-target-92hpx\" (UID: \"670c59c6-f81f-47b5-9624-2bab9d1fc847\") " pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:06:08.977982 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:08.977949 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:06:09.028031 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:09.027997 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:06:09.031389 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:09.031356 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r6hz\" (UniqueName: \"kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz\") pod \"network-check-target-92hpx\" (UID: \"670c59c6-f81f-47b5-9624-2bab9d1fc847\") " pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:06:09.108336 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:09.108255 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l4knj\"" Nov 02 22:06:09.109203 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:09.109181 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:06:09.113440 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:09.113422 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6jxx8\"" Nov 02 22:06:09.114476 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:09.114447 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:06:09.118429 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:09.118409 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g764w\"" Nov 02 22:06:09.119435 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:09.119415 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:06:09.491011 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:09.490980 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xfs82"] Nov 02 22:06:09.497861 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:06:09.497833 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae0899d_36a2_45b1_9bdf_a76acc087491.slice/crio-9f46b7dc8839c4336c856dee38954d464c38365f165f8e0bcc8b76f8a70f4d70 WatchSource:0}: Error finding container 9f46b7dc8839c4336c856dee38954d464c38365f165f8e0bcc8b76f8a70f4d70: Status 404 returned error can't find the container with id 9f46b7dc8839c4336c856dee38954d464c38365f165f8e0bcc8b76f8a70f4d70 Nov 02 22:06:09.545548 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:09.545501 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-92hpx"] Nov 02 22:06:09.546645 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:09.546624 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tp2df"] Nov 02 22:06:09.552358 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:06:09.552305 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644947bd_1d54_4a7f_afb6_463500cf17b9.slice/crio-f7f5c9078be847f40e7091278bccb917b54881298092895b9f4b7a2a1ebce01c WatchSource:0}: Error finding container f7f5c9078be847f40e7091278bccb917b54881298092895b9f4b7a2a1ebce01c: Status 404 returned error can't find the container with id f7f5c9078be847f40e7091278bccb917b54881298092895b9f4b7a2a1ebce01c Nov 02 22:06:09.553051 ip-10-0-140-221 kubenswrapper[2581]: W1102 22:06:09.553027 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670c59c6_f81f_47b5_9624_2bab9d1fc847.slice/crio-8c187516b0dff723f990a36477c64451484ea3e6bc0f9128c6335d86897275b8 WatchSource:0}: Error finding container 8c187516b0dff723f990a36477c64451484ea3e6bc0f9128c6335d86897275b8: Status 404 returned error can't find the container with id 8c187516b0dff723f990a36477c64451484ea3e6bc0f9128c6335d86897275b8 Nov 02 22:06:09.786394 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:09.786298 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:09.969550 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:09.969517 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:10.448908 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:10.448857 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-92hpx" event={"ID":"670c59c6-f81f-47b5-9624-2bab9d1fc847","Type":"ContainerStarted","Data":"8c187516b0dff723f990a36477c64451484ea3e6bc0f9128c6335d86897275b8"} Nov 02 22:06:10.450157 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:10.450128 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tp2df" event={"ID":"644947bd-1d54-4a7f-afb6-463500cf17b9","Type":"ContainerStarted","Data":"f7f5c9078be847f40e7091278bccb917b54881298092895b9f4b7a2a1ebce01c"} Nov 02 22:06:10.451411 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:10.451384 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xfs82" event={"ID":"6ae0899d-36a2-45b1-9bdf-a76acc087491","Type":"ContainerStarted","Data":"9f46b7dc8839c4336c856dee38954d464c38365f165f8e0bcc8b76f8a70f4d70"} Nov 02 22:06:10.589211 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:10.589162 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:10.776028 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:10.775941 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:11.456487 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:11.456409 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:11.661221 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:11.661128 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:12.475041 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:12.475001 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xfs82" event={"ID":"6ae0899d-36a2-45b1-9bdf-a76acc087491","Type":"ContainerStarted","Data":"cdba4978262c0a0922fbcdbefb5bacdba6e05d73629786ed1f0b3cac5b5422c6"} Nov 02 22:06:12.475540 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:12.475119 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xfs82" event={"ID":"6ae0899d-36a2-45b1-9bdf-a76acc087491","Type":"ContainerStarted","Data":"a6f456f1db5c9b8c57a5d0642087e336192d960abd894ee35cead303ac127230"} Nov 02 22:06:12.603267 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:12.603194 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xfs82" podStartSLOduration=67.539842348 podStartE2EDuration="1m9.603173979s" totalImagesPullingTime="2.063331631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:12.602109952 +0000 UTC m=+70.173975837" watchObservedRunningTime="2026-11-02 22:06:12.603173979 +0000 UTC m=+70.175039888" Nov 02 22:06:12.760498 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:12.760452 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:12.954652 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:12.954592 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:13.482545 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:13.482509 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"272c657c-1cb5-4f20-a3b7-035c535c2498","Type":"ContainerStarted","Data":"71c41332890264d27200d0629c0cb1a985229c1bcddd404c5796c5a816cdd45b"} Nov 02 22:06:13.483057 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:13.482599 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"272c657c-1cb5-4f20-a3b7-035c535c2498","Type":"ContainerStarted","Data":"b8be972e5e542e15ad9b2774d6cca6ce1d8089e98b82aa43824ee1e312852dc9"} Nov 02 22:06:13.893001 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:13.892965 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-75d4dd99d4-vfls2"] Nov 02 22:06:13.895660 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:13.895628 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:13.970205 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:13.970169 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75d4dd99d4-vfls2"] Nov 02 22:06:13.975122 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:13.975071 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2xh\" (UniqueName: \"kubernetes.io/projected/7d6c8f46-7560-45fb-8d07-f32136e14bce-kube-api-access-5p2xh\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:13.975122 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:13.975121 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-oauth-serving-cert\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:13.975347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:13.975170 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-serving-cert\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:13.975347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:13.975194 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-service-ca\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:13.975347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:13.975216 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-config\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:13.975347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:13.975251 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-trusted-ca-bundle\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:13.975347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:13.975286 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-oauth-config\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:13.975347 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:13.975310 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d6c8f46-7560-45fb-8d07-f32136e14bce-tmp\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.076044 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.076008 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-oauth-config\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.076044 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.076057 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d6c8f46-7560-45fb-8d07-f32136e14bce-tmp\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.076293 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.076103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2xh\" (UniqueName: \"kubernetes.io/projected/7d6c8f46-7560-45fb-8d07-f32136e14bce-kube-api-access-5p2xh\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.076293 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.076126 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-oauth-serving-cert\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.076293 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.076184 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-serving-cert\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.076293 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.076211 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-service-ca\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.076293 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.076239 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-config\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.076293 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.076278 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-trusted-ca-bundle\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.077097 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.077066 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d6c8f46-7560-45fb-8d07-f32136e14bce-tmp\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.077219 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.077198 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-oauth-serving-cert\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.077335 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.077317 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-service-ca\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.077419 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.077399 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-trusted-ca-bundle\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.077993 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.077968 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-config\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.080087 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.080060 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-oauth-config\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.080185 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.080150 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-serving-cert\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.137111 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.137073 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2xh\" (UniqueName: \"kubernetes.io/projected/7d6c8f46-7560-45fb-8d07-f32136e14bce-kube-api-access-5p2xh\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.211312 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.211227 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:14.491017 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.490912 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-92hpx" event={"ID":"670c59c6-f81f-47b5-9624-2bab9d1fc847","Type":"ContainerStarted","Data":"61772942c1b08265dfe7af0d638847ff7909a908aeab23741232e47ae7e8e46d"} Nov 02 22:06:14.491441 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.491022 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:06:14.604012 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.603946 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-92hpx" podStartSLOduration=67.531791878 podStartE2EDuration="1m11.603923552s" totalImagesPullingTime="4.072131674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:14.602298075 +0000 UTC m=+72.174163962" watchObservedRunningTime="2026-11-02 22:06:14.603923552 +0000 UTC m=+72.175789439" Nov 02 22:06:14.610745 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.610694 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75d4dd99d4-vfls2"] Nov 02 22:06:14.734757 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.734688 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:14.906759 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:14.906674 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:15.496928 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:15.496805 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d4dd99d4-vfls2" event={"ID":"7d6c8f46-7560-45fb-8d07-f32136e14bce","Type":"ContainerStarted","Data":"68310839cfeb8481a4b50be3fda419859d1dcc406f30ec4659c5ad362807667b"} Nov 02 22:06:15.496928 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:15.496883 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d4dd99d4-vfls2" event={"ID":"7d6c8f46-7560-45fb-8d07-f32136e14bce","Type":"ContainerStarted","Data":"85140f1873aedf163a05981c5444c605f5659b20ff33594380bb0316f5f7cbd7"} Nov 02 22:06:15.702309 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:15.702247 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-75d4dd99d4-vfls2" podStartSLOduration=2.7022249069999997 podStartE2EDuration="2.702224907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:15.660811641 +0000 UTC m=+73.232677534" watchObservedRunningTime="2026-11-02 22:06:15.702224907 +0000 UTC m=+73.274090794" Nov 02 22:06:16.505911 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:16.505804 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"272c657c-1cb5-4f20-a3b7-035c535c2498","Type":"ContainerStarted","Data":"ce02962aeb3d3f93c6b43f04e4492a69a82ee40c0c6db38fd8a519ca0fced895"} Nov 02 22:06:16.505911 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:16.505899 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"272c657c-1cb5-4f20-a3b7-035c535c2498","Type":"ContainerStarted","Data":"b4b371d4ee20c71dc80ca2263fef811167c2e4a116ede6c1c6aeab345496b0fe"} Nov 02 22:06:16.506413 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:16.505941 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"272c657c-1cb5-4f20-a3b7-035c535c2498","Type":"ContainerStarted","Data":"58308a880679d6e6a966d64f4113567f63cfacd9c4bf3e000299bb1f5cdd0399"} Nov 02 22:06:16.506413 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:16.505983 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"272c657c-1cb5-4f20-a3b7-035c535c2498","Type":"ContainerStarted","Data":"8d41805cdf51542166697800adbd4e227ddf9b40058e0c71ac1c95230a3786c2"} Nov 02 22:06:16.539359 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:16.539322 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:16.715319 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:16.715283 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:16.731780 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:16.731723 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.819375828 podStartE2EDuration="10.731699926s" totalImagesPullingTime="6.896652328s" totalInitContainerRuntime="15.67177ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:16.682607362 +0000 UTC m=+74.254473249" watchObservedRunningTime="2026-11-02 22:06:16.731699926 +0000 UTC m=+74.303565813" Nov 02 22:06:17.020546 ip-10-0-140-221 kubenswrapper[2581]: I1102 22:06:17.020507 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:17.120243 ip-10-0-140-221 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 22:06:17.135306 ip-10-0-140-221 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 22:06:17.135540 ip-10-0-140-221 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 22:06:17.135606 ip-10-0-140-221 systemd[1]: kubelet.service: Consumed 4.778s CPU time. Nov 02 22:06:17.153588 ip-10-0-140-221 systemd[1]: Starting Kubernetes Kubelet... Nov 02 22:06:17.220113 ip-10-0-140-221 kubenswrapper[5871]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:06:17.220113 ip-10-0-140-221 kubenswrapper[5871]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 22:06:17.220113 ip-10-0-140-221 kubenswrapper[5871]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:06:17.220113 ip-10-0-140-221 kubenswrapper[5871]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 22:06:17.222286 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222198 5871 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:06:17.222286 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222284 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:06:17.222286 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222287 5871 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:06:17.222286 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222290 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:06:17.222286 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222294 5871 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:06:17.222286 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222297 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222302 5871 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222308 5871 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222311 5871 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222314 5871 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222317 5871 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222320 5871 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222323 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222326 5871 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222329 5871 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222331 5871 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222334 5871 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222344 5871 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222347 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222350 5871 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222353 5871 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222356 5871 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222359 5871 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222361 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:06:17.222515 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222364 5871 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222367 5871 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222371 5871 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222374 5871 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222377 5871 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222380 5871 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222383 5871 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222385 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222388 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222391 5871 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222394 5871 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222397 5871 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222400 5871 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222403 5871 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222406 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222409 5871 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222412 5871 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222415 5871 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222418 5871 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222420 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:06:17.222991 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222423 5871 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222426 5871 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222428 5871 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222431 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222434 5871 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222437 5871 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222440 5871 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222442 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222446 5871 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222449 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222451 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222454 5871 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222457 5871 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222461 5871 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222465 5871 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222468 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222471 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222474 5871 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222477 5871 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:06:17.223459 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222480 5871 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222482 5871 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222485 5871 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222489 5871 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222492 5871 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222494 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222497 5871 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222500 5871 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222503 5871 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222506 5871 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222508 5871 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222511 5871 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222514 5871 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222517 5871 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222519 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222522 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222525 5871 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222529 5871 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222532 5871 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:06:17.223970 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222535 5871 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222538 5871 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222540 5871 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222543 5871 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222546 5871 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222548 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222551 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222554 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222556 5871 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222560 5871 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222562 5871 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222565 5871 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222567 5871 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222570 5871 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222584 5871 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222588 5871 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222593 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222597 5871 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222601 5871 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222606 5871 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:06:17.224402 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222610 5871 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222615 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222620 5871 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.222623 5871 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223430 5871 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223437 5871 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223441 5871 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223444 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223447 5871 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223450 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223453 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223457 5871 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223459 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223462 5871 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223465 5871 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223468 5871 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223471 5871 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223474 5871 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223477 5871 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:06:17.224892 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223480 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223483 5871 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223485 5871 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223488 5871 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223491 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223494 5871 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223498 5871 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223500 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223503 5871 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223506 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223508 5871 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223511 5871 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223514 5871 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223516 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223520 5871 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223523 5871 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223525 5871 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223528 5871 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223531 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:06:17.225395 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223536 5871 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223540 5871 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223543 5871 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223546 5871 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223549 5871 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223552 5871 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223556 5871 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223559 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223561 5871 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223564 5871 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223567 5871 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223570 5871 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223573 5871 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223577 5871 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223581 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223584 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223587 5871 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223590 5871 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223593 5871 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223595 5871 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:06:17.225845 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223598 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223601 5871 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223603 5871 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223606 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223609 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223612 5871 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223614 5871 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223617 5871 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223620 5871 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223623 5871 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223626 5871 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223630 5871 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223633 5871 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223636 5871 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223638 5871 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223641 5871 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223644 5871 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223647 5871 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223649 5871 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:06:17.226341 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223656 5871 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223659 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223662 5871 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223664 5871 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223667 5871 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223670 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223672 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223675 5871 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223678 5871 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223681 5871 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223684 5871 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223686 5871 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223689 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223692 5871 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223695 5871 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223698 5871 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223700 5871 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223703 5871 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223706 5871 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:06:17.226796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223709 5871 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223712 5871 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223720 5871 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223723 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223726 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223729 5871 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223732 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223735 5871 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223738 5871 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223741 5871 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223744 5871 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223746 5871 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223749 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.223751 5871 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224284 5871 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224291 5871 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224295 5871 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224298 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224301 5871 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224304 5871 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:06:17.227264 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224308 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224312 5871 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224316 5871 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224320 5871 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224323 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224326 5871 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224329 5871 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224331 5871 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224334 5871 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224337 5871 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224340 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224342 5871 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224345 5871 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224349 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224352 5871 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224354 5871 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224357 5871 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224359 5871 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224362 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:06:17.227731 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224365 5871 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224369 5871 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224372 5871 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224375 5871 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224378 5871 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224381 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224383 5871 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224387 5871 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224390 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224393 5871 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224396 5871 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224399 5871 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224401 5871 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224404 5871 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224407 5871 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224410 5871 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224412 5871 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224415 5871 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224418 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224421 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:06:17.228191 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224424 5871 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224426 5871 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224429 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224431 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224434 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224437 5871 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224440 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224442 5871 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224445 5871 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224448 5871 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224451 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224453 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224456 5871 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224459 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224461 5871 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224464 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224467 5871 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224469 5871 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224472 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:06:17.228656 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224475 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224478 5871 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224481 5871 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224483 5871 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224486 5871 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224488 5871 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224491 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224494 5871 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224496 5871 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224499 5871 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224501 5871 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224504 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224506 5871 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224509 5871 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224512 5871 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224514 5871 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224517 5871 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224520 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224522 5871 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224525 5871 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:06:17.229123 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224528 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224531 5871 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224534 5871 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224536 5871 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224539 5871 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224542 5871 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224544 5871 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224547 5871 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224549 5871 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224552 5871 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224555 5871 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224557 5871 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224560 5871 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224562 5871 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224566 5871 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224568 5871 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224571 5871 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224573 5871 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224576 5871 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224580 5871 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:06:17.229619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224582 5871 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.224585 5871 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224687 5871 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224702 5871 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224708 5871 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224713 5871 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224718 5871 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224721 5871 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224725 5871 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224733 5871 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224736 5871 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224740 5871 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224744 5871 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224748 5871 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224751 5871 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224754 5871 flags.go:64] FLAG: --cgroup-root="" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224758 5871 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224761 5871 flags.go:64] FLAG: --client-ca-file="" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224764 5871 flags.go:64] FLAG: --cloud-provider="external" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224767 5871 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224772 5871 flags.go:64] FLAG: --cluster-domain="" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224776 5871 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224779 5871 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224782 5871 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 22:06:17.230195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224786 5871 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224790 5871 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224793 5871 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224798 5871 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224802 5871 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224805 5871 flags.go:64] FLAG: --contention-profiling="false" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224808 5871 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224811 5871 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224814 5871 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224817 5871 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224822 5871 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224825 5871 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224828 5871 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224831 5871 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224834 5871 flags.go:64] FLAG: --enable-server="true" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224838 5871 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224842 5871 flags.go:64] FLAG: --event-burst="100" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224847 5871 flags.go:64] FLAG: --event-qps="50" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224850 5871 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224855 5871 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224858 5871 flags.go:64] FLAG: --eviction-hard="" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224863 5871 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224867 5871 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224882 5871 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224886 5871 flags.go:64] FLAG: --eviction-soft="" Nov 02 22:06:17.230765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224889 5871 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224892 5871 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224896 5871 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224899 5871 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224902 5871 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224905 5871 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224908 5871 flags.go:64] FLAG: --feature-gates="" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224913 5871 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224916 5871 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224919 5871 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224923 5871 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224926 5871 flags.go:64] FLAG: --healthz-port="10248" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224930 5871 flags.go:64] FLAG: --help="false" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224933 5871 flags.go:64] FLAG: --hostname-override="ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224937 5871 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224940 5871 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224943 5871 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224947 5871 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224950 5871 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224953 5871 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224956 5871 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224960 5871 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224963 5871 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 22:06:17.231388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224966 5871 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224972 5871 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224975 5871 flags.go:64] FLAG: --kube-reserved="" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224980 5871 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224983 5871 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224987 5871 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224990 5871 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224993 5871 flags.go:64] FLAG: --lock-file="" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224996 5871 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.224999 5871 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225003 5871 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225009 5871 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225012 5871 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225015 5871 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225018 5871 flags.go:64] FLAG: --logging-format="text" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225021 5871 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225025 5871 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225028 5871 flags.go:64] FLAG: --manifest-url="" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225031 5871 flags.go:64] FLAG: --manifest-url-header="" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225036 5871 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225039 5871 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225044 5871 flags.go:64] FLAG: --max-pods="110" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225048 5871 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225051 5871 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225054 5871 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 22:06:17.231946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225057 5871 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225061 5871 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225064 5871 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225067 5871 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225074 5871 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225078 5871 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225081 5871 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225084 5871 flags.go:64] FLAG: --pod-cidr="" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225089 5871 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225092 5871 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225097 5871 flags.go:64] FLAG: --pods-per-core="0" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225101 5871 flags.go:64] FLAG: --port="10250" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225104 5871 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225107 5871 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-03af722d6dfd167a6" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225110 5871 flags.go:64] FLAG: --qos-reserved="" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225114 5871 flags.go:64] FLAG: --read-only-port="10255" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225117 5871 flags.go:64] FLAG: --register-node="true" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225120 5871 flags.go:64] FLAG: --register-with-taints="" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225124 5871 flags.go:64] FLAG: --registry-burst="10" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225127 5871 flags.go:64] FLAG: --registry-qps="5" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225130 5871 flags.go:64] FLAG: --reserved-cpus="" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225133 5871 flags.go:64] FLAG: --reserved-memory="" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225137 5871 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225141 5871 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225144 5871 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225147 5871 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 22:06:17.232531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225150 5871 flags.go:64] FLAG: --runonce="false" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225154 5871 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225157 5871 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225161 5871 flags.go:64] FLAG: --seccomp-default="false" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225164 5871 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225167 5871 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225171 5871 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225174 5871 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225177 5871 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225180 5871 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225183 5871 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225187 5871 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225189 5871 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225193 5871 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225205 5871 flags.go:64] FLAG: --system-cgroups="" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225210 5871 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225217 5871 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225220 5871 flags.go:64] FLAG: --tls-cert-file="" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225223 5871 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225229 5871 flags.go:64] FLAG: --tls-min-version="" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225232 5871 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225235 5871 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225238 5871 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225241 5871 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225244 5871 flags.go:64] FLAG: --v="2" Nov 02 22:06:17.233155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225253 5871 flags.go:64] FLAG: --version="false" Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225258 5871 flags.go:64] FLAG: --vmodule="" Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225262 5871 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225266 5871 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225420 5871 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225426 5871 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225429 5871 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225432 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225435 5871 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225437 5871 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225440 5871 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225444 5871 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225447 5871 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225450 5871 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225453 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225456 5871 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225458 5871 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225461 5871 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225464 5871 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225467 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:06:17.233763 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225470 5871 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225474 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225477 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225481 5871 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225484 5871 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225487 5871 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225490 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225493 5871 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225495 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225499 5871 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225501 5871 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225504 5871 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225507 5871 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225510 5871 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225513 5871 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225516 5871 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225518 5871 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225522 5871 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225524 5871 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225527 5871 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:06:17.234248 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225530 5871 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225533 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225535 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225538 5871 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225541 5871 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225544 5871 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225546 5871 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225549 5871 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225552 5871 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225554 5871 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225557 5871 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225560 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225563 5871 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225567 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225569 5871 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225574 5871 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225577 5871 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225580 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225582 5871 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:06:17.234709 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225585 5871 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225588 5871 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225590 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225593 5871 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225596 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225598 5871 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225601 5871 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225604 5871 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225606 5871 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225609 5871 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225612 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225615 5871 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225617 5871 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225620 5871 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225623 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225627 5871 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225631 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225634 5871 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225637 5871 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:06:17.235334 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225640 5871 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225642 5871 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225645 5871 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225648 5871 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225650 5871 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225653 5871 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225656 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225660 5871 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225663 5871 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225667 5871 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225670 5871 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225672 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225675 5871 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225678 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225681 5871 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225683 5871 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225687 5871 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225692 5871 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225695 5871 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:06:17.236119 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225698 5871 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:06:17.236866 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225700 5871 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:06:17.236866 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225703 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:06:17.236866 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225706 5871 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:06:17.236866 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225709 5871 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:06:17.236866 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225711 5871 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:06:17.236866 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225714 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:06:17.236866 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225716 5871 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:06:17.236866 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225719 5871 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:06:17.236866 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225722 5871 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:06:17.236866 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225725 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:06:17.236866 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225727 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:06:17.236866 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225730 5871 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:06:17.236866 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.225756 5871 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:06:17.236866 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225794 5871 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:06:17.236866 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225799 5871 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:06:17.236866 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225802 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225805 5871 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225808 5871 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225811 5871 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225814 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225819 5871 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225824 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225828 5871 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225831 5871 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225833 5871 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225836 5871 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225839 5871 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225842 5871 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225844 5871 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225847 5871 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225850 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225853 5871 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225856 5871 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225858 5871 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:06:17.237337 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225861 5871 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225864 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225867 5871 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225883 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225887 5871 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225891 5871 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225897 5871 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225903 5871 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225908 5871 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225912 5871 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225917 5871 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225921 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225924 5871 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225926 5871 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225929 5871 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225932 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225935 5871 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225937 5871 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225942 5871 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:06:17.237813 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225946 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225949 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225952 5871 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225955 5871 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225957 5871 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225960 5871 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225963 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225966 5871 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225968 5871 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225971 5871 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225974 5871 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225977 5871 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225979 5871 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225982 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225985 5871 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225988 5871 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225990 5871 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225993 5871 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225996 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.225998 5871 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:06:17.238425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226001 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226004 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226006 5871 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226009 5871 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226012 5871 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226015 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226018 5871 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226021 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226024 5871 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226027 5871 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226030 5871 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226035 5871 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226040 5871 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226043 5871 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226046 5871 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226049 5871 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226052 5871 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226054 5871 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226057 5871 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226060 5871 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:06:17.238941 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226063 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226066 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226068 5871 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226071 5871 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226074 5871 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226077 5871 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226079 5871 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226082 5871 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226086 5871 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226089 5871 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226091 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226094 5871 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226097 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226099 5871 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226102 5871 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226105 5871 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226107 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226110 5871 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226113 5871 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:06:17.239476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226116 5871 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226120 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226122 5871 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226125 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226130 5871 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226134 5871 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.226136 5871 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.230358 5871 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.230379 5871 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.230399 5871 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.230404 5871 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230468 5871 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230474 5871 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230478 5871 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230481 5871 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230484 5871 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230487 5871 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230491 5871 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230494 5871 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:06:17.239999 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230497 5871 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230500 5871 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230502 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230505 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230508 5871 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230511 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230513 5871 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230516 5871 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230520 5871 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230525 5871 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230529 5871 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230533 5871 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230536 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230540 5871 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230543 5871 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230546 5871 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230549 5871 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230552 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230555 5871 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:06:17.240502 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230557 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230560 5871 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230569 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230572 5871 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230575 5871 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230578 5871 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230580 5871 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230583 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230586 5871 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230589 5871 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230592 5871 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230594 5871 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230597 5871 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230600 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230603 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230606 5871 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230609 5871 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230612 5871 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230614 5871 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230617 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:06:17.241018 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230620 5871 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230622 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230625 5871 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230628 5871 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230630 5871 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230633 5871 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230636 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230639 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230642 5871 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230645 5871 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230647 5871 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230650 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230653 5871 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230655 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230658 5871 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230667 5871 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230669 5871 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230672 5871 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230675 5871 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230677 5871 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:06:17.241620 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230680 5871 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230683 5871 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230685 5871 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230688 5871 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230690 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230693 5871 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230696 5871 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230698 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230701 5871 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230704 5871 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230707 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230709 5871 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230712 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230715 5871 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230718 5871 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230720 5871 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230723 5871 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230726 5871 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230729 5871 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230732 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:06:17.242155 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230735 5871 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230738 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230740 5871 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230743 5871 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230746 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230748 5871 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230751 5871 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230753 5871 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230761 5871 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230764 5871 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230767 5871 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230770 5871 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230773 5871 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230776 5871 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230779 5871 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230782 5871 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230785 5871 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230787 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:06:17.242628 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.230790 5871 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:06:17.243076 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.230820 5871 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:06:17.243076 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231013 5871 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:06:17.243076 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231018 5871 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:06:17.243076 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231021 5871 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:06:17.243076 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231024 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:06:17.243076 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231027 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:06:17.243076 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231029 5871 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:06:17.243076 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231032 5871 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:06:17.243076 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231035 5871 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:06:17.243076 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231038 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:06:17.243076 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231041 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:06:17.243076 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231044 5871 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:06:17.243076 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231046 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:06:17.243076 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231049 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:06:17.243076 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231052 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231055 5871 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231058 5871 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231060 5871 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231063 5871 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231066 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231069 5871 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231071 5871 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231080 5871 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231083 5871 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231085 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231088 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231091 5871 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231093 5871 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231096 5871 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231099 5871 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231102 5871 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231105 5871 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231108 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231110 5871 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:06:17.243462 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231113 5871 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231115 5871 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231118 5871 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231121 5871 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231124 5871 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231126 5871 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231129 5871 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231131 5871 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231134 5871 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231137 5871 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231140 5871 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231143 5871 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231145 5871 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231148 5871 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231151 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231153 5871 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231156 5871 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231159 5871 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231161 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231164 5871 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:06:17.243948 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231167 5871 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231177 5871 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231180 5871 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231182 5871 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231185 5871 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231189 5871 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231194 5871 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231197 5871 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231201 5871 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231206 5871 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231209 5871 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231212 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231215 5871 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231217 5871 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231220 5871 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231223 5871 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231226 5871 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231228 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231231 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:06:17.244429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231233 5871 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231236 5871 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231239 5871 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231247 5871 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231250 5871 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231252 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231255 5871 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231258 5871 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231261 5871 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231263 5871 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231266 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231269 5871 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231271 5871 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231274 5871 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231276 5871 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231284 5871 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231287 5871 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231290 5871 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231292 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:06:17.244937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231295 5871 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:06:17.245421 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231297 5871 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:06:17.245421 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231300 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:06:17.245421 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231303 5871 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:06:17.245421 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231306 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:06:17.245421 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231308 5871 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:06:17.245421 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231311 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:06:17.245421 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231314 5871 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:06:17.245421 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231316 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:06:17.245421 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231319 5871 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:06:17.245421 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231321 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:06:17.245421 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231324 5871 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:06:17.245421 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231327 5871 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:06:17.245421 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231330 5871 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:06:17.245421 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231332 5871 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:06:17.245421 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.231356 5871 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 22:06:17.245421 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231393 5871 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231398 5871 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231402 5871 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231406 5871 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231409 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231412 5871 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231414 5871 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231417 5871 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231420 5871 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231422 5871 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231425 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231428 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231430 5871 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231441 5871 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231444 5871 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231447 5871 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231450 5871 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231453 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231456 5871 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 22:06:17.245802 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231458 5871 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231461 5871 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231463 5871 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231466 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231469 5871 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231471 5871 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231474 5871 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231477 5871 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231479 5871 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231482 5871 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231484 5871 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231487 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231489 5871 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231492 5871 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231495 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231497 5871 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231500 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231502 5871 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231505 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231508 5871 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 22:06:17.246271 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231510 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231513 5871 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231515 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231518 5871 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231520 5871 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231523 5871 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231526 5871 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231534 5871 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231537 5871 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231539 5871 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231542 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231545 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231548 5871 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231550 5871 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231553 5871 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231556 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231558 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231561 5871 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231563 5871 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 22:06:17.246750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231566 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231569 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231571 5871 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231574 5871 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231576 5871 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231579 5871 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231583 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231585 5871 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231588 5871 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231590 5871 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231593 5871 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231596 5871 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231599 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231601 5871 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231604 5871 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231606 5871 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231609 5871 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231612 5871 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231614 5871 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 22:06:17.247293 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231617 5871 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231620 5871 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231629 5871 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231632 5871 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231635 5871 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231637 5871 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231640 5871 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231642 5871 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231645 5871 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231648 5871 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231650 5871 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231653 5871 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231655 5871 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231658 5871 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231660 5871 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231663 5871 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231666 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231668 5871 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231674 5871 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231676 5871 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 22:06:17.247799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231679 5871 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 22:06:17.248298 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231681 5871 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 22:06:17.248298 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231684 5871 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 22:06:17.248298 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231686 5871 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 22:06:17.248298 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231689 5871 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 22:06:17.248298 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231692 5871 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 22:06:17.248298 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231694 5871 feature_gate.go:386] unrecognized feature gate: Example Nov 02 22:06:17.248298 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231698 5871 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 22:06:17.248298 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:17.231702 5871 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 22:06:17.248298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.231785 5871 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 22:06:17.248298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.232843 5871 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 22:06:17.248298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.232924 5871 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 22:06:17.248298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.233139 5871 server.go:1014] "Starting client certificate rotation" Nov 02 22:06:17.248298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.233272 5871 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 22:06:17.248298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.233314 5871 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 22:00:02 +0000 UTC" deadline="2028-07-16 07:15:37.926752748 +0000 UTC" Nov 02 22:06:17.248298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.233359 5871 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14913h9m20.693397259s" Nov 02 22:06:17.248298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.233689 5871 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:06:17.248723 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.235040 5871 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 22:06:17.248723 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.238288 5871 log.go:25] "Validated CRI v1 runtime API" Nov 02 22:06:17.248723 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.240286 5871 log.go:25] "Validated CRI v1 image API" Nov 02 22:06:17.248723 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.241030 5871 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 22:06:17.248723 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.243442 5871 fs.go:135] Filesystem UUIDs: map[0d690c39-df18-469b-a300-46c8be12f295:/dev/nvme0n1p3 79757a5c-3a20-4ae0-9226-5f2097252f39:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 02 22:06:17.248901 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.243460 5871 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/052c047e99f57ddbabb7d762bf82fd0f0574338e430f070867180cc7f2dcb417/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/052c047e99f57ddbabb7d762bf82fd0f0574338e430f070867180cc7f2dcb417/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16b570a4eb4094d0ad6ec1911e23a79c1fd097ff8f379536f786416721565d99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16b570a4eb4094d0ad6ec1911e23a79c1fd097ff8f379536f786416721565d99/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25e3292d0c67e6de4f616aafcfb0b0051f79b82ca336b9839f160f9a4298a8e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25e3292d0c67e6de4f616aafcfb0b0051f79b82ca336b9839f160f9a4298a8e6/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b5ae94787a66232e1c33984629a0a7e02c459370657d27f42c5cd9b38c17823/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b5ae94787a66232e1c33984629a0a7e02c459370657d27f42c5cd9b38c17823/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30d3ea3093c8ab907dacae103f38bbc7f21c76d106337a775fe667dbba6f670c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30d3ea3093c8ab907dacae103f38bbc7f21c76d106337a775fe667dbba6f670c/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/363b3db111d77b5a6fc5447bdc646228edf2e548ef89f06617c4ade0a6a00c4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/363b3db111d77b5a6fc5447bdc646228edf2e548ef89f06617c4ade0a6a00c4a/userdata/shm major:0 minor:285 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46409b905adf9a74d7a4145f7092b909ace522c82b374060c92da1474e1ce7f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46409b905adf9a74d7a4145f7092b909ace522c82b374060c92da1474e1ce7f8/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/501126d919c6c9ab589aecebf1bbab791fb4ff4249ddde3f7ee12e171df7c5f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/501126d919c6c9ab589aecebf1bbab791fb4ff4249ddde3f7ee12e171df7c5f6/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/688a334dbdb09643b4eeffbd2800ec19eb51db27b196952cc2361e0226cd28e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/688a334dbdb09643b4eeffbd2800ec19eb51db27b196952cc2361e0226cd28e7/userdata/shm major:0 minor:318 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f2763a0e9d2236e0b2f4fe07cc15e3892b302fb52acc8d26e4bbf345c6c2ea9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f2763a0e9d2236e0b2f4fe07cc15e3892b302fb52acc8d26e4bbf345c6c2ea9/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/777fade67cb5c42aa8ec5cbf133c5f79591fdd26c88efc67ee3689d67ffd47d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/777fade67cb5c42aa8ec5cbf133c5f79591fdd26c88efc67ee3689d67ffd47d4/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/794a5fcae42ade03d3565c050f435fdfc9f71feaa020200de7b8456b89148471/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/794a5fcae42ade03d3565c050f435fdfc9f71feaa020200de7b8456b89148471/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85140f1873aedf163a05981c5444c605f5659b20ff33594380bb0316f5f7cbd7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85140f1873aedf163a05981c5444c605f5659b20ff33594380bb0316f5f7cbd7/userdata/shm major:0 minor:460 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c187516b0dff723f990a36477c64451484ea3e6bc0f9128c6335d86897275b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c187516b0dff723f990a36477c64451484ea3e6bc0f9128c6335d86897275b8/userdata/shm major:0 minor:421 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90ce1faee2a613e6e6b1dd0ffa46235eb09b18377f9bf507c45d583f3efff9ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90ce1faee2a613e6e6b1dd0ffa46235eb09b18377f9bf507c45d583f3efff9ed/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/991641b7f1220a32684d82210fd0bf79fd19056167b58b18957d88d2973c815c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/991641b7f1220a32684d82210fd0bf79fd19056167b58b18957d88d2973c815c/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d10e0153a84c93325ad46da510a95e83f03b8e16b45a623938c384724c6bb8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d10e0153a84c93325ad46da510a95e83f03b8e16b45a623938c384724c6bb8e/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f46b7dc8839c4336c856dee38954d464c38365f165f8e0bcc8b76f8a70f4d70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f46b7dc8839c4336c856dee38954d464c38365f165f8e0bcc8b76f8a70f4d70/userdata/shm major:0 minor:417 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5c2101529a22c04fa525c591d93dc1f5982dfcf7a252b9603227560914c4208/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5c2101529a22c04fa525c591d93dc1f5982dfcf7a252b9603227560914c4208/userdata/shm major:0 minor:291 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a72e21182b1155c0e19ae1737ceb9e5537968ab3a3246bd4dbe19a5c590214ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a72e21182b1155c0e19ae1737ceb9e5537968ab3a3246bd4dbe19a5c590214ed/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0ff9b530e22aee46bd8c080adf8af0cc217c8634220388e791d49352153bf1f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0ff9b530e22aee46bd8c080adf8af0cc217c8634220388e791d49352153bf1f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be03c1c4bf106b4e190ff5861d9f12fb41fd9db1b7bbca2dc2ab18a2bee7bdca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be03c1c4bf106b4e190ff5861d9f12fb41fd9db1b7bbca2dc2ab18a2bee7bdca/userdata/shm major:0 minor:405 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c750bbf4e947a072895ce3eb34c9a418999b007f53c73d462c9cc8ad410dfff5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c750bbf4e947a072895ce3eb34c9a418999b007f53c73d462c9cc8ad410dfff5/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d11690e1d9b5e4668765d0d4c930146f950b078d877671ac7e94053623d6c028/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d11690e1d9b5e4668765d0d4c930146f950b078d877671ac7e94053623d6c028/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7f5c9078be847f40e7091278bccb917b54881298092895b9f4b7a2a1ebce01c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7f5c9078be847f40e7091278bccb917b54881298092895b9f4b7a2a1ebce01c/userdata/shm major:0 minor:419 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2/volumes/kubernetes.io~projected/kube-api-access-c2ph7:{mountpoint:/var/lib/kubelet/pods/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2/volumes/kubernetes.io~projected/kube-api-access-c2ph7 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volume-subpaths/web-config/config-reloader/2 major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volume-subpaths/web-config/prometheus/15 major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~empty-dir/config-out major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~projected/kube-api-access-dbw9d:{mountpoint:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~projected/kube-api-access-dbw9d major:0 minor:404 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~projected/tls-assets major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/config major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:403 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/web-config major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f6652ca-37d9-45f8-a135-19dc4cc60fc3/volumes/kubernetes.io~projected/kube-api-access-qzwll:{mountpoint:/var/lib/kubelet/pods/2f6652ca-37d9-45f8-a135-19dc4cc60fc3/volumes/kubernetes.io~projected/kube-api-access-qzwll major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb97b10-9112-4dd4-8d7d-a2c864c39358/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2fb97b10-9112-4dd4-8d7d-a2c864c39358/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb97b10-9112-4dd4-8d7d-a2c864c39358/volumes/kubernetes.io~projected/kube-api-access-bsphb:{mountpoint:/var/lib/kubelet/pods/2fb97b10-9112-4dd4-8d7d-a2c864c39358/volumes/kubernetes.io~projected/kube-api-access-bsphb major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fb97b10-9112-4dd4-8d7d-a2c864c39358/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2fb97b10-9112-4dd4-8d7d-a2c864c39358/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/425cd6ec-68c1-48ca-8243-b15082ae9553/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/425cd6ec-68c1-48ca-8243-b15082ae9553/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3/volumes/kubernetes.io~projected/kube-api-access-npbhr:{mountpoint:/var/lib/kubelet/pods/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3/volumes/kubernetes.io~projected/kube-api-access-npbhr major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/644947bd-1d54-4a7f-afb6-463500cf17b9/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/644947bd-1d54-4a7f-afb6-463500cf17b9/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:415 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/670c59c6-f81f-47b5-9624-2bab9d1fc847/volumes/kubernetes.io~projected/kube-api-access-4r6hz:{mountpoint:/var/lib/kubelet/pods/670c59c6-f81f-47b5-9624-2bab9d1fc847/volumes/kubernetes.io~projected/kube-api-access-4r6hz major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67edaf75-8924-48b9-9a81-93089ab7c82d/volumes/kubernetes.io~projected/kube-api-access-cfxq4:{mountpoint:/var/lib/kubelet/pods/67edaf75-8924-48b9-9a81-93089ab7c82d/volumes/kubernetes.io~projected/kube-api-access-cfxq4 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67edaf75-8924-48b9-9a81-93089ab7c82d/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/67edaf75-8924-48b9-9a81-93089ab7c82d/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ae0899d-36a2-45b1-9bdf-a76acc087491/volumes/kubernetes.io~projected/kube-api-access-2jzj5:{mountpoint:/var/lib/kubelet/pods/6ae0899d-36a2-45b1-9bdf-a76acc087491/volumes/kubernetes.io~projected/kube-api-access-2jzj5 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ae0899d-36a2-45b1-9bdf-a76acc087491/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/6ae0899d-36a2-45b1-9bdf-a76acc087491/volumes/kubernetes.io~secret/metrics-certs major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70d481aa-8880-41b2-957a-285385bceebd/volumes/kubernetes.io~projected/kube-api-access-bzqmd:{mountpoint:/var/lib/kubelet/pods/70d481aa-8880-41b2-957a-285385bceebd/volumes/kubernetes.io~projected/kube-api-access-bzqmd major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70d481aa-8880-41b2-957a-285385bceebd/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/70d481aa-8880-41b2-957a-285385bceebd/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/759f8e6e-92ee-4b83-9447-949cb34a2cd1/volumes/kubernetes.io~projected/kube-api-access-4xbbh:{mountpoint:/var/lib/kubelet/pods/759f8e6e-92ee-4b83-9447-949cb34a2cd1/volumes/kubernetes.io~projected/kube-api-access-4xbbh major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/759f8e6e-92ee-4b83-9447-949cb34a2cd1/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/759f8e6e-92ee-4b83-9447-949cb34a2cd1/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/759f8e6e-92ee-4b83-9447-949cb34a2cd1/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/759f8e6e-92ee-4b83-9447-949cb34a2cd1/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b199a04-e491-43c9-baf3-0d9e6c663dfa/volumes/kubernetes.io~projected/kube-api-access-vjx2f:{mountpoint:/var/lib/kubelet/pods/7b199a04-e491-43c9-baf3-0d9e6c663dfa/volumes/kubernetes.io~projected/kube-api-access-vjx2f major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b199a04-e491-43c9-baf3-0d9e6c663dfa/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/7b199a04-e491-43c9-baf3-0d9e6c663dfa/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b199a04-e491-43c9-baf3-0d9e6c663dfa/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/7b199a04-e491-43c9-baf3-0d9e6c663dfa/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b199a04-e491-43c9-baf3-0d9e6c663dfa/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/7b199a04-e491-43c9-baf3-0d9e6c663dfa/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d6c8f46-7560-45fb-8d07-f32136e14bce/volumes/kubernetes.io~projected/kube-api-access-5p2xh:{mountpoint:/var/lib/kubelet/pods/7d6c8f46-7560-45fb-8d07-f32136e14bce/volumes/kubernetes.io~projected/kube-api-access-5p2xh major:0 minor:459 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d6c8f46-7560-45fb-8d07-f32136e14bce/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/7d6c8f46-7560-45fb-8d07-f32136e14bce/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d6c8f46-7560-45fb-8d07-f32136e14bce/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/7d6c8f46-7560-45fb-8d07-f32136e14bce/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:458 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1/volumes/kubernetes.io~projected/kube-api-access-8nbpz:{mountpoint:/var/lib/kubelet/pods/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1/volumes/kubernetes.io~projected/kube-api-access-8nbpz major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa26f0e-38c8-4705-b64a-0a943508b307/volumes/kubernetes.io~projected/kube-api-access-m6g87:{mountpoint:/var/lib/kubelet/pods/8aa26f0e-38c8-4705-b64a-0a943508b307/volumes/kubernetes.io~projected/kube-api-access-m6g87 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b8a9a43-5fd2-4e49-b313-9ff58be7524b/volumes/kubernetes.io~projected/kube-api-access-pn52n:{mountpoint:/var/lib/kubelet/pods/8b8a9a43-5fd2-4e49-b313-9ff58be7524b/volumes/kubernetes.io~projected/kube-api-access-pn52n major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volume-subpaths/web-config/alertmanager/9 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volume-subpaths/web-config/config-reloader/6 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~empty-dir/config-out major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~projected/kube-api-access-4sql9:{mountpoint:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~projected/kube-api-access-4sql9 major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~projected/tls-assets major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/config-volume major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/web-config major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b44243f9-55cd-40b4-87e8-53913b4c26c4/volumes/kubernetes.io~projected/kube-api-access-qgwzb:{mountpoint:/var/lib/kubelet/pods/b44243f9-55cd-40b4-87e8-53913b4c26c4/volumes/kubernetes.io~projected/kube-api-access-qgwzb major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba6a658f-a491-458f-a3be-2d71c676c069/volumes/kubernetes.io~projected/kube-api-access-w9b2t:{mountpoint:/var/lib/kubelet/pods/ba6a658f-a491-458f-a3be-2d71c676c069/volumes/kubernetes.io~projected/kube-api-access-w9b2t major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba6a658f-a491-458f-a3be-2d71c676c069/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/ba6a658f-a491-458f-a3be-2d71c676c069/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be31cb75-89a7-4b84-b26f-2050acfa249c/volumes/kubernetes.io~projected/kube-api-access-bz6cr:{mountpoint:/var/lib/kubelet/pods/be31cb75-89a7-4b84-b26f-2050acfa249c/volumes/kubernetes.io~projected/kube-api-access-bz6cr major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d725e391-7af5-4691-b327-259e86639f8e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d725e391-7af5-4691-b327-259e86639f8e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d725e391-7af5-4691-b327-259e86639f8e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d725e391-7af5-4691-b327-259e86639f8e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d725e391-7af5-4691-b327-259e86639f8e/volumes/kubernetes.io~projected/kube-api-access-xf6jx:{mountpoint:/var/lib/kubelet/pods/d725e391-7af5-4691-b327-259e86639f8e/volumes/kubernetes.io~projected/kube-api-access-xf6jx major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e95934d1-f5df-4c06-ab39-5e7c8d58858b/volumes/kubernetes.io~projected/kube-api-access-gbnh2:{mountpoint:/var/lib/kubelet/pods/e95934d1-f5df-4c06-ab39-5e7c8d58858b/volumes/kubernetes.io~projected/kube-api-access-gbnh2 major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e95934d1-f5df-4c06-ab39-5e7c8d58858b/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/e95934d1-f5df-4c06-ab39-5e7c8d58858b/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e95934d1-f5df-4c06-ab39-5e7c8d58858b/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/e95934d1-f5df-4c06-ab39-5e7c8d58858b/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec8310e5-d852-44c1-901b-b54a6c13a798/volumes/kubernetes.io~projected/kube-api-access-5gwv8:{mountpoint:/var/lib/kubelet/pods/ec8310e5-d852-44c1-901b-b54a6c13a798/volumes/kubernetes.io~projected/kube-api-access-5gwv8 major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec8310e5-d852-44c1-901b-b54a6c13a798/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ec8310e5-d852-44c1-901b-b54a6c13a798/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec8310e5-d852-44c1-901b-b54a6c13a798/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ec8310e5-d852-44c1-901b-b54a6c13a798/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:281 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4cf75f716f4327182c0dcc6b657e6d55890601e7e28837e3dc174e7cb3289087/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-103:{mountpoint:/var/lib/containers/storage/overlay/1bf6c66d15ae65e0edcf01eda73a0260295f33a8c94cbce7d3cda384aa71d339/merged major:0 minor:103 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/24ffc7171947afaa1a54fc8fc6a8b30247c71c3f3b3ec4bb43f6e5a16b847bcc/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/6733c813201b92ff2545d219d43d4e92b93e82858394e4d822045667879b4f20/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ab30e65dfefecf0532da1fe180cf87c107f5d6446e9fdd3c5c6b65e4b65e4329/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2719d7fb379e65999318101c59c8bb676588774940ea056c5c91dc931436a914/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/7c08139cebfb6b65cdde3dcd5c61e120de29ac7b85d97c08037d76ff429bf204/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/904a4bcbe7fda8b865fb06f6e9b08969dd3f2bf23dd7539c58fa1ae5739927a4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/23f790cb23e01c5eff3cacaa130b4d83020c6a444fa88300632376a5b57b14b2/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/decdd9bdf5e123990ac8545724faa4561e00984be3123e7048f69bc4d073a6f3/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/ddb51d1d34769bcde0f58ec9e3b6b71568162ce082190ecf3ead9d935113755f/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/4bd937fe5c545f1f30a1cdbcbfca8e78d9849ab3c4b606a00bd065fc67798c8d/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/d4120de4211599c9054460c54ecf84f90010575f9c495a064b7d5f74683ec0f0/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9b2bbb72ddb17fa029922e864805e6e23270d285d03ada78109524a1e8b2fdbe/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f3fceb542b4b1861ec4179044a9155d640699f0c9ffc1f707092319355b8fef3/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/42b21e3e4930a55d440b323ddb2b12cdd464e682d24ecc2a1c7d88abc3dc5ab2/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/85de33099d295d3dd07d904ee2c174afb9a789ee1b64872ecd355b0f1e63b267/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/fb8f36299886dcfec917c000560a0118ed965f535a8ca993cf7bbec21633c684/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b18bfbbf8d0e0c0b73939593354e9636be207f8db146852e863fb9fc8536362f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b9568c0d20135b9b31519d4b7d57f1c0131b1792c2bf99ad9dcc6cd7727bc3d3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/5a024d860961d7879eb9a619a435819c51f7a2e93792449f1abf906e64b31612/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/07316be685921bca16fa0021aad062be20d68a5eda995f8f7c0f4bd02762f036/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/1aef10268716c1c847f171500ab7a70e767caa14d592fa8b48719ed8af87d137/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/941063a67ea0d142197623823506dbc29b67eb70339d289ffc6d397404a844f4/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/eb7ccd80e650572681cd85f2f4cc7550ae7152891ed9cfe314035e54fafa8936/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/08de06164c8724f5d6b1efcc67422cd3f2f010c3b29d1b0cd7d128017f676b2a/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/cc2c33f8b137d9074068c03300a39cc4ac687f137160e825483176fc097d6f17/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/d65065ae900243a7456859cefb227cbe185040043c5627958cdf0a6245d9e3ee/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/bf273930a93a510f4d503cc54660ee18130610948a8e20b23bc233ec0b6e5f81/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/90fb5494d6a1f3641954615deb649e0fde1fdead0e9accb7374e34b2f5205959/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/a6525742f6564f2a141d12609f2b8a6bb29804ddb87ced320fabb0fdaa2d3039/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/038e09dd193abc95a39f4f826a2d694510945e511443afdb4f7c438c3bda82dc/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/78d3867bce5b84e3f8f3a0b831fad494bd0ec432f5074e6b83b72cb423750bf6/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/70065b14062d271806d623a33790106dec6d947fffb38dada62d880107789fd7/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/94cea9eb7b6f0e39584d0fb656fe5d0bb395fdc78fc2d7e5347c1309f4349753/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/6acfc0eefae7b16c56f89ac573de0b8b17d8a590bcffceeaf205fcf958948736/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/77bb7225783ce09cfba0755c7264a81eb84b449df1812d305d9131b487c45a93/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/a3cadb61bfa2517ad6b273d974484829e904d5fc4dcca8b697382b51e920dc1e/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/af4ee44e6296ea6d584bced4300a9de30e6d6e981f6b222f2d407f39f304e7a1/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/17849d1f363fa6de75cfb782b005e582dd9172e98b82742c875a2f088d35b737/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/a5df1250eb5a4260caebd836ab61248f8dfb803004bda12dfddde284c8c9193d/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/9e233af8e4482757b931acf54f0a463ac3b7bea73c04e932f04d681beb030307/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/3334d53ef7a46391fc6ddb6939e3a7f140c76b204153c5974a3191dcaecb3b83/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/20abf16cb45c5ef299e42941c7b042d59277d86122d41a14bd149908cf2ddbce/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/ff72096e660ff75f08aabce4386a5920961c8cedaf37c94d138865fa8d760e0f/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/ff38393c143a72ed5e3630ba730ce0e70ec30ab4a9afb2e83f8b8776d9cb70f7/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/739798549ee85e87d2af93fae1ad1871e4431e4480fab34958ebdcb00cd0c79a/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/430657d625d1f0ee00f47492c294b131e81a0efce9e7579ffc1a3dd16e649431/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/d28307e3a514f120e0ec5783c415c6a3324384743c4e1b64a3d3efa605d28d98/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/cb2ad66d2b9030dcf16c7ed5268ad122c0b2b56d2f06932c383ebe3142df7ee3/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/333a752936e128118081a4edbbf3d5d1d2c43b37e441e1b59a55c5da74e3e63d/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/aad6d67e64d3ac10b2d5cbe419336699d1c94207a0b3d8de58547beaca68b581/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/0e4057dfc6b69ebb68013198eaeab7a6eef65c629d0b9cc3c2b714ebde60557c/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/2e6acb17012b95e9b711e1fa2c84034c49bb7acd7c24df8a954228838dea9113/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/07270d901aa8ed60363b308e6bf6c868ab8b40c7dc678a678b6bda5952aee878/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/648798faf900df6e4cc390fc643b05675387436f6000a01810d6b799f39b9d9e/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/740cea86c4421c641320085fe97b465581a34c30a5ce5f6519ab7a1aa17ae775/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/407cb4532aefdc55ea5a0526b53deabc79e97033b3fbf53e4dc5ebb97522bf10/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/4c6aee59af4bce0f2e91741ee98448acb1ceb92422ff5bbc70b4f035e5038106/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/3221ae9b201ea558523a1c10c03db7163435f203c1b28e20a1e8d472bb776a6e/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/75c259abcc356700d0e73092a662315dce2c9d40504c1a1fea5d282558a61e5f/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/ec047643b28db8b8b0f4dc02ae43bab1176a2ead8a144cac46906943259b99f5/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-448:{mountpoint:/var/lib/containers/storage/overlay/58354b247d9f237144c9e025405ed645b56e26eddabec86388ff8faba11a3a07/merged major:0 minor:448 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/7c9b2d5f0e6c4326330db5b23617d99ed4703c24cff84b494509ff23d6b1c5fc/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-464:{mountpoint:/var/lib/containers/storage/overlay/ea87f551314a2420b28d8befbc5f7ca2cf13171c7589f928b4bbf31be9f40178/merged major:0 minor:464 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/a9d480093a7c0a012cb6b16cf4668db7db96df2497b04865f18d85e72dc1b823/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-472:{mountpoint:/var/lib/containers/storage/overlay/ca72ec064cc5cac95ba1b5ee4f0c23b52ad3ecab475446417659abdc5ec53a51/merged major:0 minor:472 fsType:overlay blockSize:0} overlay_0-477:{mountpoint:/var/lib/containers/storage/overlay/856e70cbaad88e71b4681e81d44ad353ac9cf3d35aa40839a8adf7f50bc726d7/merged major:0 minor:477 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/9639c49d44facd66b5dd6199a484b9e9bf4cd21750e62f3d64cdd8c638d01185/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-487:{mountpoint:/var/lib/containers/storage/overlay/3e782c2ed368e1ae8ae95c1924a66ed76dde5cc273b4e2222c12583f30813950/merged major:0 minor:487 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/92ddfa2b0ed5434062a5c252c336362e2aab092073f6d0b9a4f3c340326448b7/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/55a8d8dff3834a928bafa6fe8a6f3808aee4ceb1e7d091442aa014999eb15f49/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/848973de969c47b7988b284a83d5bf9690246871e8e930025de5b46daca0b9f6/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/6c43b9bfe5e2ac2f3a4f30b9e559b1049b171b61b5cae3e1da30bf0ca4fec0b0/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/bd4a2767a7432ec8a491f2d820e695928761f311a2b56268208eb97492df1d30/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/b8021c50fbeda59b6f87f244a7328263a1887b434b53967f7a3596577327766a/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/4afb1e80567b4d9d8586e0ef09b340c43e1bae615dd2baff7ebf37f161560384/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/4eb72e82b67deb96c05a68052120de9aca471657b65a910137cfce82de6aba3b/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/39be53361d1f7eb17f599d012f01f9c50ed6edfc80f35199989c5624735f5be4/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/bb9f64698fb84f4b2d2226c853495c6c151c293c99d0abe64e40dff700633dbd/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 22:06:17.253059 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.252578 5871 manager.go:217] Machine: {Timestamp:2026-11-02 22:06:17.252057439 +0000 UTC m=+0.064387608 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3127222 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23bd2316f94f86057c183258a0336f SystemUUID:ec23bd23-16f9-4f86-057c-183258a0336f BootID:67df9700-f71d-470e-a256-7bdade58927e Filesystems:[{Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/363b3db111d77b5a6fc5447bdc646228edf2e548ef89f06617c4ade0a6a00c4a/userdata/shm DeviceMajor:0 DeviceMinor:285 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b199a04-e491-43c9-baf3-0d9e6c663dfa/volumes/kubernetes.io~projected/kube-api-access-vjx2f DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25e3292d0c67e6de4f616aafcfb0b0051f79b82ca336b9839f160f9a4298a8e6/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/759f8e6e-92ee-4b83-9447-949cb34a2cd1/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:277 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:394 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e95934d1-f5df-4c06-ab39-5e7c8d58858b/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:257 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec8310e5-d852-44c1-901b-b54a6c13a798/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:272 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/777fade67cb5c42aa8ec5cbf133c5f79591fdd26c88efc67ee3689d67ffd47d4/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb97b10-9112-4dd4-8d7d-a2c864c39358/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:396 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:313 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30d3ea3093c8ab907dacae103f38bbc7f21c76d106337a775fe667dbba6f670c/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/991641b7f1220a32684d82210fd0bf79fd19056167b58b18957d88d2973c815c/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1/volumes/kubernetes.io~projected/kube-api-access-8nbpz DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:311 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/670c59c6-f81f-47b5-9624-2bab9d1fc847/volumes/kubernetes.io~projected/kube-api-access-4r6hz DeviceMajor:0 DeviceMinor:416 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8b8a9a43-5fd2-4e49-b313-9ff58be7524b/volumes/kubernetes.io~projected/kube-api-access-pn52n DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/794a5fcae42ade03d3565c050f435fdfc9f71feaa020200de7b8456b89148471/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba6a658f-a491-458f-a3be-2d71c676c069/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/052c047e99f57ddbabb7d762bf82fd0f0574338e430f070867180cc7f2dcb417/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f6652ca-37d9-45f8-a135-19dc4cc60fc3/volumes/kubernetes.io~projected/kube-api-access-qzwll DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb97b10-9112-4dd4-8d7d-a2c864c39358/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/501126d919c6c9ab589aecebf1bbab791fb4ff4249ddde3f7ee12e171df7c5f6/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b199a04-e491-43c9-baf3-0d9e6c663dfa/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:355 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d10e0153a84c93325ad46da510a95e83f03b8e16b45a623938c384724c6bb8e/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/70d481aa-8880-41b2-957a-285385bceebd/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e95934d1-f5df-4c06-ab39-5e7c8d58858b/volumes/kubernetes.io~projected/kube-api-access-gbnh2 DeviceMajor:0 DeviceMinor:259 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c187516b0dff723f990a36477c64451484ea3e6bc0f9128c6335d86897275b8/userdata/shm DeviceMajor:0 DeviceMinor:421 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a72e21182b1155c0e19ae1737ceb9e5537968ab3a3246bd4dbe19a5c590214ed/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67edaf75-8924-48b9-9a81-93089ab7c82d/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec8310e5-d852-44c1-901b-b54a6c13a798/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:281 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:398 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85140f1873aedf163a05981c5444c605f5659b20ff33594380bb0316f5f7cbd7/userdata/shm DeviceMajor:0 DeviceMinor:460 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba6a658f-a491-458f-a3be-2d71c676c069/volumes/kubernetes.io~projected/kube-api-access-w9b2t DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:244 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46409b905adf9a74d7a4145f7092b909ace522c82b374060c92da1474e1ce7f8/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d6c8f46-7560-45fb-8d07-f32136e14bce/volumes/kubernetes.io~projected/kube-api-access-5p2xh DeviceMajor:0 DeviceMinor:459 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-464 DeviceMajor:0 DeviceMinor:464 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa26f0e-38c8-4705-b64a-0a943508b307/volumes/kubernetes.io~projected/kube-api-access-m6g87 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d6c8f46-7560-45fb-8d07-f32136e14bce/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:458 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70d481aa-8880-41b2-957a-285385bceebd/volumes/kubernetes.io~projected/kube-api-access-bzqmd DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-472 DeviceMajor:0 DeviceMinor:472 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-477 DeviceMajor:0 DeviceMinor:477 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d725e391-7af5-4691-b327-259e86639f8e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d725e391-7af5-4691-b327-259e86639f8e/volumes/kubernetes.io~projected/kube-api-access-xf6jx DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16b570a4eb4094d0ad6ec1911e23a79c1fd097ff8f379536f786416721565d99/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:395 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:399 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2fb97b10-9112-4dd4-8d7d-a2c864c39358/volumes/kubernetes.io~projected/kube-api-access-bsphb DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67edaf75-8924-48b9-9a81-93089ab7c82d/volumes/kubernetes.io~projected/kube-api-access-cfxq4 DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:280 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:390 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d725e391-7af5-4691-b327-259e86639f8e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:402 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:403 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d6c8f46-7560-45fb-8d07-f32136e14bce/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:453 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:243 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:305 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:309 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:400 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-487 DeviceMajor:0 DeviceMinor:487 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3/volumes/kubernetes.io~projected/kube-api-access-npbhr DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:315 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:398 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:398 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c750bbf4e947a072895ce3eb34c9a418999b007f53c73d462c9cc8ad410dfff5/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-103 DeviceMajor:0 DeviceMinor:103 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:401 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/425cd6ec-68c1-48ca-8243-b15082ae9553/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90ce1faee2a613e6e6b1dd0ffa46235eb09b18377f9bf507c45d583f3efff9ed/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/759f8e6e-92ee-4b83-9447-949cb34a2cd1/volumes/kubernetes.io~projected/kube-api-access-4xbbh DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b44243f9-55cd-40b4-87e8-53913b4c26c4/volumes/kubernetes.io~projected/kube-api-access-qgwzb DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ec8310e5-d852-44c1-901b-b54a6c13a798/volumes/kubernetes.io~projected/kube-api-access-5gwv8 DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:305 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6ae0899d-36a2-45b1-9bdf-a76acc087491/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:411 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~projected/kube-api-access-dbw9d DeviceMajor:0 DeviceMinor:404 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-448 DeviceMajor:0 DeviceMinor:448 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f2763a0e9d2236e0b2f4fe07cc15e3892b302fb52acc8d26e4bbf345c6c2ea9/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b199a04-e491-43c9-baf3-0d9e6c663dfa/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:353 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f46b7dc8839c4336c856dee38954d464c38365f165f8e0bcc8b76f8a70f4d70/userdata/shm DeviceMajor:0 DeviceMinor:417 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b5ae94787a66232e1c33984629a0a7e02c459370657d27f42c5cd9b38c17823/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6ae0899d-36a2-45b1-9bdf-a76acc087491/volumes/kubernetes.io~projected/kube-api-access-2jzj5 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:398 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d11690e1d9b5e4668765d0d4c930146f950b078d877671ac7e94053623d6c028/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0ff9b530e22aee46bd8c080adf8af0cc217c8634220388e791d49352153bf1f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e95934d1-f5df-4c06-ab39-5e7c8d58858b/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/759f8e6e-92ee-4b83-9447-949cb34a2cd1/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b199a04-e491-43c9-baf3-0d9e6c663dfa/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:354 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/be31cb75-89a7-4b84-b26f-2050acfa249c/volumes/kubernetes.io~projected/kube-api-access-bz6cr DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5c2101529a22c04fa525c591d93dc1f5982dfcf7a252b9603227560914c4208/userdata/shm DeviceMajor:0 DeviceMinor:291 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:397 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be03c1c4bf106b4e190ff5861d9f12fb41fd9db1b7bbca2dc2ab18a2bee7bdca/userdata/shm DeviceMajor:0 DeviceMinor:405 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2/volumes/kubernetes.io~projected/kube-api-access-c2ph7 DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~projected/kube-api-access-4sql9 DeviceMajor:0 DeviceMinor:317 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/688a334dbdb09643b4eeffbd2800ec19eb51db27b196952cc2361e0226cd28e7/userdata/shm DeviceMajor:0 DeviceMinor:318 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/644947bd-1d54-4a7f-afb6-463500cf17b9/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:415 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7f5c9078be847f40e7091278bccb917b54881298092895b9f4b7a2a1ebce01c/userdata/shm DeviceMajor:0 DeviceMinor:419 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:052c047e99f57dd MacAddress:4e:af:5a:cd:99:ca Speed:10000 Mtu:8901} {Name:16b570a4eb4094d MacAddress:66:b0:6d:22:2e:bd Speed:10000 Mtu:8901} {Name:25e3292d0c67e6d MacAddress:22:a0:48:92:d9:f5 Speed:10000 Mtu:8901} {Name:363b3db111d77b5 MacAddress:e6:d3:85:1b:43:2d Speed:10000 Mtu:8901} {Name:46409b905adf9a7 MacAddress:1a:29:58:a9:bb:8d Speed:10000 Mtu:8901} {Name:688a334dbdb0964 MacAddress:d6:10:97:00:01:77 Speed:10000 Mtu:8901} {Name:6f2763a0e9d2236 MacAddress:4e:e2:61:7c:6d:6b Speed:10000 Mtu:8901} {Name:85140f1873aedf1 MacAddress:96:d1:a4:64:f3:c6 Speed:10000 Mtu:8901} {Name:8c187516b0dff72 MacAddress:66:c9:9b:50:53:fc Speed:10000 Mtu:8901} {Name:991641b7f1220a3 MacAddress:ba:77:07:13:1a:fe Speed:10000 Mtu:8901} {Name:9f46b7dc8839c43 MacAddress:8e:14:d8:6f:1e:b5 Speed:10000 Mtu:8901} {Name:a5c2101529a22c0 MacAddress:96:07:dd:eb:d8:a5 Speed:10000 Mtu:8901} {Name:be03c1c4bf106b4 MacAddress:da:22:bc:6e:10:7c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:eb:79:0c:55 Speed:0 Mtu:9001} {Name:br-int MacAddress:1e:14:15:4f:37:b8 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:eb:79:0c:55 Speed:0 Mtu:9001} {Name:f7f5c9078be847f MacAddress:2a:8d:dd:81:9e:ec Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:2e:56:6e:ce:3d:be Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:9e:d5:b8:1d:ea:a6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 22:06:17.253059 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.253062 5871 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 22:06:17.253404 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.253121 5871 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 22:06:17.253540 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.253512 5871 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 22:06:17.253695 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.253543 5871 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-221.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 22:06:17.253737 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.253698 5871 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 22:06:17.253737 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.253706 5871 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 22:06:17.253737 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.253720 5871 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:06:17.253737 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.253734 5871 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 22:06:17.253851 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.253843 5871 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 22:06:17.254097 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.254086 5871 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 22:06:17.254228 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.254219 5871 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 22:06:17.254266 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.254258 5871 kubelet.go:493] "Attempting to sync node with API server" Nov 02 22:06:17.254295 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.254271 5871 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 22:06:17.254295 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.254282 5871 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 22:06:17.254295 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.254293 5871 kubelet.go:405] "Adding apiserver pod source" Nov 02 22:06:17.254418 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.254302 5871 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 22:06:17.255362 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.255347 5871 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 22:06:17.255415 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.255362 5871 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:06:17.255990 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.255968 5871 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 22:06:17.260179 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260159 5871 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 22:06:17.260179 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260174 5871 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 22:06:17.260335 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260274 5871 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 22:06:17.260479 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260467 5871 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 22:06:17.260510 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260497 5871 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 22:06:17.260605 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260596 5871 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 22:06:17.260634 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260610 5871 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 22:06:17.260634 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260617 5871 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 22:06:17.260634 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260622 5871 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 22:06:17.260634 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260629 5871 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 22:06:17.260634 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260634 5871 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 22:06:17.260784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260640 5871 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 22:06:17.260784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260646 5871 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 22:06:17.260784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260653 5871 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 22:06:17.260784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260659 5871 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 22:06:17.260784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260668 5871 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 22:06:17.260784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260678 5871 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 22:06:17.260784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260701 5871 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 22:06:17.260784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.260707 5871 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 22:06:17.264326 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.264310 5871 server.go:1263] "Started kubelet" Nov 02 22:06:17.264761 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.264429 5871 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 22:06:17.264761 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.264464 5871 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 22:06:17.264761 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.264538 5871 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 22:06:17.264949 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.264822 5871 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 22:06:17.264906 ip-10-0-140-221 systemd[1]: Started Kubernetes Kubelet. Nov 02 22:06:17.266135 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.266033 5871 server.go:317] "Adding debug handlers to kubelet server" Nov 02 22:06:17.268443 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.268424 5871 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 22:06:17.268545 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.268462 5871 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 22:06:17.269937 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.268542 5871 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:00:04 +0000 UTC" deadline="2028-07-27 08:22:28.633308432 +0000 UTC" Nov 02 22:06:17.269937 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.268583 5871 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15178h16m11.364728221s" Nov 02 22:06:17.269937 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.268628 5871 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 22:06:17.269937 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.268635 5871 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 22:06:17.269937 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:17.268715 5871 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-221.us-west-2.compute.internal\" not found" Nov 02 22:06:17.269937 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.268753 5871 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 22:06:17.270239 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.270166 5871 factory.go:55] Registering systemd factory Nov 02 22:06:17.270239 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.270186 5871 factory.go:223] Registration of the systemd container factory successfully Nov 02 22:06:17.271033 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.270753 5871 factory.go:153] Registering CRI-O factory Nov 02 22:06:17.271033 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.270798 5871 factory.go:223] Registration of the crio container factory successfully Nov 02 22:06:17.271033 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.270902 5871 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 22:06:17.271033 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.270926 5871 factory.go:103] Registering Raw factory Nov 02 22:06:17.271033 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.270943 5871 manager.go:1199] Started watching for new ooms in manager Nov 02 22:06:17.273236 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.273213 5871 manager.go:319] Starting recovery of all containers Nov 02 22:06:17.277865 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.277822 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f6652ca-37d9-45f8-a135-19dc4cc60fc3" volumeName="kubernetes.io/configmap/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-cni-binary-copy" seLinuxMountContext="" Nov 02 22:06:17.277977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.277904 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d6c8f46-7560-45fb-8d07-f32136e14bce" volumeName="kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-service-ca" seLinuxMountContext="" Nov 02 22:06:17.277977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.277925 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" volumeName="kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 02 22:06:17.277977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.277946 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" volumeName="kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-cluster-tls-config" seLinuxMountContext="" Nov 02 22:06:17.277977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.277961 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be31cb75-89a7-4b84-b26f-2050acfa249c" volumeName="kubernetes.io/projected/be31cb75-89a7-4b84-b26f-2050acfa249c-kube-api-access-bz6cr" seLinuxMountContext="" Nov 02 22:06:17.277977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.277975 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d725e391-7af5-4691-b327-259e86639f8e" volumeName="kubernetes.io/empty-dir/d725e391-7af5-4691-b327-259e86639f8e-tmp" seLinuxMountContext="" Nov 02 22:06:17.278218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.277993 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95934d1-f5df-4c06-ab39-5e7c8d58858b" volumeName="kubernetes.io/empty-dir/e95934d1-f5df-4c06-ab39-5e7c8d58858b-tmp" seLinuxMountContext="" Nov 02 22:06:17.278218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.278005 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="644947bd-1d54-4a7f-afb6-463500cf17b9" volumeName="kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret" seLinuxMountContext="" Nov 02 22:06:17.278218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.278027 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d6c8f46-7560-45fb-8d07-f32136e14bce" volumeName="kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-oauth-config" seLinuxMountContext="" Nov 02 22:06:17.278218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.278040 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba6a658f-a491-458f-a3be-2d71c676c069" volumeName="kubernetes.io/secret/ba6a658f-a491-458f-a3be-2d71c676c069-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 22:06:17.278218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.278058 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95934d1-f5df-4c06-ab39-5e7c8d58858b" volumeName="kubernetes.io/projected/e95934d1-f5df-4c06-ab39-5e7c8d58858b-kube-api-access-gbnh2" seLinuxMountContext="" Nov 02 22:06:17.278218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.278074 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="759f8e6e-92ee-4b83-9447-949cb34a2cd1" volumeName="kubernetes.io/secret/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 22:06:17.278218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.278087 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 22:06:17.278218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.278105 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-metrics-client-certs" seLinuxMountContext="" Nov 02 22:06:17.278218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.278123 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d481aa-8880-41b2-957a-285385bceebd" volumeName="kubernetes.io/secret/70d481aa-8880-41b2-957a-285385bceebd-cert" seLinuxMountContext="" Nov 02 22:06:17.278218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.278140 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-web-config" seLinuxMountContext="" Nov 02 22:06:17.278218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.278184 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" volumeName="kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 02 22:06:17.279228 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.278209 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b44243f9-55cd-40b4-87e8-53913b4c26c4" volumeName="kubernetes.io/configmap/b44243f9-55cd-40b4-87e8-53913b4c26c4-iptables-alerter-script" seLinuxMountContext="" Nov 02 22:06:17.279312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279243 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f3ab9d6-1afa-499c-80d2-a1d11981eeb2" volumeName="kubernetes.io/configmap/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-metrics-client-ca" seLinuxMountContext="" Nov 02 22:06:17.279312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279263 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f3ab9d6-1afa-499c-80d2-a1d11981eeb2" volumeName="kubernetes.io/projected/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-api-access-c2ph7" seLinuxMountContext="" Nov 02 22:06:17.279312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279276 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-grpc-tls" seLinuxMountContext="" Nov 02 22:06:17.279312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279288 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f6652ca-37d9-45f8-a135-19dc4cc60fc3" volumeName="kubernetes.io/configmap/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-daemon-config" seLinuxMountContext="" Nov 02 22:06:17.279312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279299 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3" volumeName="kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 22:06:17.279312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279312 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-config" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279323 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279334 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" volumeName="kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279345 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d6c8f46-7560-45fb-8d07-f32136e14bce" volumeName="kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-serving-cert" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279356 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d725e391-7af5-4691-b327-259e86639f8e" volumeName="kubernetes.io/empty-dir/d725e391-7af5-4691-b327-259e86639f8e-etc-tuned" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279373 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb97b10-9112-4dd4-8d7d-a2c864c39358" volumeName="kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-env-overrides" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279383 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" volumeName="kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-oauth-serving-cert" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279395 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba6a658f-a491-458f-a3be-2d71c676c069" volumeName="kubernetes.io/configmap/ba6a658f-a491-458f-a3be-2d71c676c069-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279407 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f3ab9d6-1afa-499c-80d2-a1d11981eeb2" volumeName="kubernetes.io/configmap/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279417 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb97b10-9112-4dd4-8d7d-a2c864c39358" volumeName="kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovnkube-script-lib" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279427 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d6c8f46-7560-45fb-8d07-f32136e14bce" volumeName="kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-trusted-ca-bundle" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279440 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b199a04-e491-43c9-baf3-0d9e6c663dfa" volumeName="kubernetes.io/configmap/7b199a04-e491-43c9-baf3-0d9e6c663dfa-metrics-server-audit-profiles" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279452 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d6c8f46-7560-45fb-8d07-f32136e14bce" volumeName="kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-oauth-serving-cert" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279465 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d725e391-7af5-4691-b327-259e86639f8e" volumeName="kubernetes.io/projected/d725e391-7af5-4691-b327-259e86639f8e-kube-api-access-xf6jx" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279480 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279495 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-config-out" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279509 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb97b10-9112-4dd4-8d7d-a2c864c39358" volumeName="kubernetes.io/projected/2fb97b10-9112-4dd4-8d7d-a2c864c39358-kube-api-access-bsphb" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279524 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="759f8e6e-92ee-4b83-9447-949cb34a2cd1" volumeName="kubernetes.io/projected/759f8e6e-92ee-4b83-9447-949cb34a2cd1-kube-api-access-4xbbh" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279538 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="759f8e6e-92ee-4b83-9447-949cb34a2cd1" volumeName="kubernetes.io/empty-dir/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-textfile" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279552 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b199a04-e491-43c9-baf3-0d9e6c663dfa" volumeName="kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-client-ca-bundle" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279565 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" volumeName="kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279576 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95934d1-f5df-4c06-ab39-5e7c8d58858b" volumeName="kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-serving-cert" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279594 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-kube-api-access-dbw9d" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279603 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb97b10-9112-4dd4-8d7d-a2c864c39358" volumeName="kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovnkube-config" seLinuxMountContext="" Nov 02 22:06:17.279601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279611 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be31cb75-89a7-4b84-b26f-2050acfa249c" volumeName="kubernetes.io/empty-dir/be31cb75-89a7-4b84-b26f-2050acfa249c-tmp-dir" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279624 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95934d1-f5df-4c06-ab39-5e7c8d58858b" volumeName="kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-config" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279635 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec8310e5-d852-44c1-901b-b54a6c13a798" volumeName="kubernetes.io/configmap/ec8310e5-d852-44c1-901b-b54a6c13a798-metrics-client-ca" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279654 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec8310e5-d852-44c1-901b-b54a6c13a798" volumeName="kubernetes.io/secret/ec8310e5-d852-44c1-901b-b54a6c13a798-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279662 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d6c8f46-7560-45fb-8d07-f32136e14bce" volumeName="kubernetes.io/projected/7d6c8f46-7560-45fb-8d07-f32136e14bce-kube-api-access-5p2xh" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279671 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b8a9a43-5fd2-4e49-b313-9ff58be7524b" volumeName="kubernetes.io/projected/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-kube-api-access-pn52n" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279681 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec8310e5-d852-44c1-901b-b54a6c13a798" volumeName="kubernetes.io/secret/ec8310e5-d852-44c1-901b-b54a6c13a798-openshift-state-metrics-tls" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279693 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f3ab9d6-1afa-499c-80d2-a1d11981eeb2" volumeName="kubernetes.io/secret/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-tls" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279702 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="425cd6ec-68c1-48ca-8243-b15082ae9553" volumeName="kubernetes.io/configmap/425cd6ec-68c1-48ca-8243-b15082ae9553-konnectivity-ca" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279715 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d6c8f46-7560-45fb-8d07-f32136e14bce" volumeName="kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-config" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279745 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" volumeName="kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-serving-cert" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279754 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b44243f9-55cd-40b4-87e8-53913b4c26c4" volumeName="kubernetes.io/projected/b44243f9-55cd-40b4-87e8-53913b4c26c4-kube-api-access-qgwzb" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279763 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f3ab9d6-1afa-499c-80d2-a1d11981eeb2" volumeName="kubernetes.io/empty-dir/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-volume-directive-shadow" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279770 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67edaf75-8924-48b9-9a81-93089ab7c82d" volumeName="kubernetes.io/configmap/67edaf75-8924-48b9-9a81-93089ab7c82d-config-volume" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279778 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b199a04-e491-43c9-baf3-0d9e6c663dfa" volumeName="kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279785 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="425cd6ec-68c1-48ca-8243-b15082ae9553" volumeName="kubernetes.io/secret/425cd6ec-68c1-48ca-8243-b15082ae9553-agent-certs" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279792 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="759f8e6e-92ee-4b83-9447-949cb34a2cd1" volumeName="kubernetes.io/configmap/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279800 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b199a04-e491-43c9-baf3-0d9e6c663dfa" volumeName="kubernetes.io/empty-dir/7b199a04-e491-43c9-baf3-0d9e6c663dfa-audit-log" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279808 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b199a04-e491-43c9-baf3-0d9e6c663dfa" volumeName="kubernetes.io/projected/7b199a04-e491-43c9-baf3-0d9e6c663dfa-kube-api-access-vjx2f" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279816 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95934d1-f5df-4c06-ab39-5e7c8d58858b" volumeName="kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-oauth-config" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279823 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-metrics-client-ca" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279831 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279838 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3" volumeName="kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cni-binary-copy" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279845 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3" volumeName="kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279852 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" volumeName="kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279858 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95934d1-f5df-4c06-ab39-5e7c8d58858b" volumeName="kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-oauth-serving-cert" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279866 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279884 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67edaf75-8924-48b9-9a81-93089ab7c82d" volumeName="kubernetes.io/projected/67edaf75-8924-48b9-9a81-93089ab7c82d-kube-api-access-cfxq4" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279891 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" volumeName="kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-out" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279899 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" volumeName="kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-volume" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279907 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-db" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279914 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ae0899d-36a2-45b1-9bdf-a76acc087491" volumeName="kubernetes.io/projected/6ae0899d-36a2-45b1-9bdf-a76acc087491-kube-api-access-2jzj5" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279921 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" volumeName="kubernetes.io/empty-dir/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-tmp" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279929 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3" volumeName="kubernetes.io/projected/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-kube-api-access-npbhr" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279939 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b199a04-e491-43c9-baf3-0d9e6c663dfa" volumeName="kubernetes.io/configmap/7b199a04-e491-43c9-baf3-0d9e6c663dfa-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279947 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" volumeName="kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-service-ca" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279956 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" volumeName="kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-main-db" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279965 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba6a658f-a491-458f-a3be-2d71c676c069" volumeName="kubernetes.io/projected/ba6a658f-a491-458f-a3be-2d71c676c069-kube-api-access-w9b2t" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279972 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279981 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f6652ca-37d9-45f8-a135-19dc4cc60fc3" volumeName="kubernetes.io/projected/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-kube-api-access-qzwll" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279990 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="670c59c6-f81f-47b5-9624-2bab9d1fc847" volumeName="kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.279998 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d481aa-8880-41b2-957a-285385bceebd" volumeName="kubernetes.io/projected/70d481aa-8880-41b2-957a-285385bceebd-kube-api-access-bzqmd" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280005 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" volumeName="kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-kube-api-access-4sql9" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280012 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba6a658f-a491-458f-a3be-2d71c676c069" volumeName="kubernetes.io/empty-dir/ba6a658f-a491-458f-a3be-2d71c676c069-data-volume" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280020 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280028 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67edaf75-8924-48b9-9a81-93089ab7c82d" volumeName="kubernetes.io/empty-dir/67edaf75-8924-48b9-9a81-93089ab7c82d-tmp-dir" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280035 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" volumeName="kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-oauth-config" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280043 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa26f0e-38c8-4705-b64a-0a943508b307" volumeName="kubernetes.io/configmap/8aa26f0e-38c8-4705-b64a-0a943508b307-serviceca" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280050 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" volumeName="kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-config" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280058 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec8310e5-d852-44c1-901b-b54a6c13a798" volumeName="kubernetes.io/projected/ec8310e5-d852-44c1-901b-b54a6c13a798-kube-api-access-5gwv8" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280066 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280074 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa26f0e-38c8-4705-b64a-0a943508b307" volumeName="kubernetes.io/projected/8aa26f0e-38c8-4705-b64a-0a943508b307-kube-api-access-m6g87" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280080 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" volumeName="kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-metrics-client-ca" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280089 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-tls-assets" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280097 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="759f8e6e-92ee-4b83-9447-949cb34a2cd1" volumeName="kubernetes.io/configmap/759f8e6e-92ee-4b83-9447-949cb34a2cd1-metrics-client-ca" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280104 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95934d1-f5df-4c06-ab39-5e7c8d58858b" volumeName="kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-trusted-ca-bundle" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280112 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67edaf75-8924-48b9-9a81-93089ab7c82d" volumeName="kubernetes.io/secret/67edaf75-8924-48b9-9a81-93089ab7c82d-metrics-tls" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280120 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" volumeName="kubernetes.io/projected/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-kube-api-access-8nbpz" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280127 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280135 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280142 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="759f8e6e-92ee-4b83-9447-949cb34a2cd1" volumeName="kubernetes.io/secret/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-tls" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280149 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" volumeName="kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-tls-assets" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280156 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e95934d1-f5df-4c06-ab39-5e7c8d58858b" volumeName="kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-service-ca" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280164 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="272c657c-1cb5-4f20-a3b7-035c535c2498" volumeName="kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280174 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fb97b10-9112-4dd4-8d7d-a2c864c39358" volumeName="kubernetes.io/secret/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280197 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ae0899d-36a2-45b1-9bdf-a76acc087491" volumeName="kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280205 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b199a04-e491-43c9-baf3-0d9e6c663dfa" volumeName="kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-secret-metrics-server-tls" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280213 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d6c8f46-7560-45fb-8d07-f32136e14bce" volumeName="kubernetes.io/empty-dir/7d6c8f46-7560-45fb-8d07-f32136e14bce-tmp" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280220 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" volumeName="kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-web-config" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280228 5871 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f3ab9d6-1afa-499c-80d2-a1d11981eeb2" volumeName="kubernetes.io/secret/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280235 5871 reconstruct.go:97] "Volume reconstruction finished" Nov 02 22:06:17.280659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.280253 5871 reconciler.go:29] "Reconciler: start to sync state" Nov 02 22:06:17.304311 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.303911 5871 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 22:06:17.305437 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.305411 5871 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 22:06:17.305437 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.305439 5871 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 22:06:17.305623 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.305462 5871 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 22:06:17.305623 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:17.305505 5871 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 22:06:17.317498 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.317465 5871 generic.go:356] "Generic (PLEG): container finished" podID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerID="368273d4a42d7eae3ebb350aa7b1a269bcedd1b8a57eefb77aa54f03d4f5d6d1" exitCode=0 Nov 02 22:06:17.334173 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.334138 5871 generic.go:356] "Generic (PLEG): container finished" podID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerID="5064217c1727c131b0e57008af6162977bb22e38ebaf8db195cb216cf2f4ee4d" exitCode=0 Nov 02 22:06:17.342548 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.342514 5871 generic.go:356] "Generic (PLEG): container finished" podID="44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3" containerID="5967fcec3a0871fdda577534ce21d58fd8cfbbe282c38b64c0e7e2e658e34d9b" exitCode=0 Nov 02 22:06:17.342548 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.342543 5871 generic.go:356] "Generic (PLEG): container finished" podID="44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3" containerID="5f973736c678ae54333a86eccc8c51102a1c7cfc53959c9692a391020fb50c12" exitCode=0 Nov 02 22:06:17.342548 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.342552 5871 generic.go:356] "Generic (PLEG): container finished" podID="44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3" containerID="57bf7e0c1cc20fabf4068af3e5672f02c0aa7a8176abe3c905486cda5c1588bc" exitCode=0 Nov 02 22:06:17.342793 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.342563 5871 generic.go:356] "Generic (PLEG): container finished" podID="44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3" containerID="8b860c812745c401ce78678f8c1f82cad9c0ce297d3af926f4047e45d2f43897" exitCode=0 Nov 02 22:06:17.342793 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.342570 5871 generic.go:356] "Generic (PLEG): container finished" podID="44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3" containerID="014bbf9e271fa69c5ab298a2c7e15af22906259b557ea37be8256917f139c903" exitCode=0 Nov 02 22:06:17.342793 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.342577 5871 generic.go:356] "Generic (PLEG): container finished" podID="44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3" containerID="e39358ebbd94827a980d1949af3d31fd9358d9faec3ab07773a4e04451e09fa6" exitCode=0 Nov 02 22:06:17.344602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.344566 5871 generic.go:356] "Generic (PLEG): container finished" podID="c91a35388f9dd8d39adce6d4d82b0bac" containerID="d2f7cdaaeea9512c6b237657365664e2bed7a5e85435fd140d7ead330646aa22" exitCode=0 Nov 02 22:06:17.368973 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:17.368938 5871 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-221.us-west-2.compute.internal\" not found" Nov 02 22:06:17.369629 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.369598 5871 generic.go:356] "Generic (PLEG): container finished" podID="759f8e6e-92ee-4b83-9447-949cb34a2cd1" containerID="769540672d397b834678476587d94a7646470538b21f4acc7687782638914785" exitCode=0 Nov 02 22:06:17.392106 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.392039 5871 manager.go:324] Recovery completed Nov 02 22:06:17.405622 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:17.405583 5871 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 22:06:17.416620 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.416594 5871 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:06:17.418660 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.418636 5871 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:06:17.418745 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.418675 5871 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:06:17.418745 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.418690 5871 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:06:17.424209 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.424184 5871 cpu_manager.go:226] "Starting" policy="none" Nov 02 22:06:17.424209 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.424201 5871 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 22:06:17.424209 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.424204 5871 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:06:17.424388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.424225 5871 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 22:06:17.424388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.424358 5871 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 22:06:17.424388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.424368 5871 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 22:06:17.424388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.424387 5871 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 22:06:17.424526 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.424392 5871 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 22:06:17.424526 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.424398 5871 policy_none.go:50] "Start" Nov 02 22:06:17.424526 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.424408 5871 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 22:06:17.424526 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.424417 5871 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 22:06:17.425216 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.425189 5871 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:06:17.425271 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.425232 5871 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:06:17.425271 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.425247 5871 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:06:17.425867 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.425854 5871 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 22:06:17.425926 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.425887 5871 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 22:06:17.425926 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.425895 5871 policy_none.go:44] "Start" Nov 02 22:06:17.430859 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.430826 5871 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 22:06:17.436222 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.436206 5871 manager.go:341] "Starting Device Plugin manager" Nov 02 22:06:17.436304 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:17.436241 5871 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 22:06:17.436304 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.436254 5871 server.go:87] "Starting device plugin registration server" Nov 02 22:06:17.436591 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.436579 5871 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 22:06:17.436626 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.436594 5871 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 22:06:17.436709 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.436691 5871 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 22:06:17.436844 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.436805 5871 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 22:06:17.436969 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.436852 5871 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 22:06:17.436969 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.436859 5871 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 22:06:17.437058 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.436918 5871 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T22:06:17.436839727Z","UUID":"3511e946-d0f5-42a8-b3f8-1be2f537bb15","Handler":null,"Name":"","Endpoint":""} Nov 02 22:06:17.438666 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.438637 5871 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 22:06:17.438666 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.438658 5871 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 22:06:17.438836 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:17.438741 5871 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 22:06:17.438836 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:17.438792 5871 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-221.us-west-2.compute.internal\" not found" Nov 02 22:06:17.454747 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.454714 5871 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:06:17.454987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.454755 5871 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:06:17.454987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.454815 5871 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 22:06:17.500832 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.500800 5871 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:06:17.537695 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.537653 5871 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 22:06:17.544378 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.544352 5871 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 22:06:17.544509 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.544387 5871 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 22:06:17.544509 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.544400 5871 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-221.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 22:06:17.544509 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.544484 5871 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:17.547125 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.547103 5871 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 22:06:17.606420 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.606360 5871 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal"] Nov 02 22:06:17.607168 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.607135 5871 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:17.607288 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.607167 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" event={"ID":"c91a35388f9dd8d39adce6d4d82b0bac","Type":"ContainerStarted","Data":"67cc4224298aabcc53ae23f53cc40e031891b0ac62c4363297e5f9a8828667c6"} Nov 02 22:06:17.607288 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.607211 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" event={"ID":"c91a35388f9dd8d39adce6d4d82b0bac","Type":"ContainerDied","Data":"d2f7cdaaeea9512c6b237657365664e2bed7a5e85435fd140d7ead330646aa22"} Nov 02 22:06:17.607288 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.607231 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" event={"ID":"c91a35388f9dd8d39adce6d4d82b0bac","Type":"ContainerStarted","Data":"2b5ae94787a66232e1c33984629a0a7e02c459370657d27f42c5cd9b38c17823"} Nov 02 22:06:17.607288 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.607248 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" event={"ID":"8332e3a50281a723271a56acbbdc782f","Type":"ContainerStarted","Data":"826c346915a554ac42e7d8030e5969ffb37e500d4b980df7b58014fd15972282"} Nov 02 22:06:17.607288 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.607265 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" event={"ID":"8332e3a50281a723271a56acbbdc782f","Type":"ContainerStarted","Data":"777fade67cb5c42aa8ec5cbf133c5f79591fdd26c88efc67ee3689d67ffd47d4"} Nov 02 22:06:17.607452 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.607420 5871 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:17.654489 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.654405 5871 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:17.654489 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.654489 5871 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:17.669185 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.669150 5871 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:06:17.669345 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:17.669220 5871 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:17.669345 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.669150 5871 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:06:17.669345 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:17.669301 5871 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:17.757341 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.757303 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8332e3a50281a723271a56acbbdc782f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"8332e3a50281a723271a56acbbdc782f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:17.757341 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.757341 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c91a35388f9dd8d39adce6d4d82b0bac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"c91a35388f9dd8d39adce6d4d82b0bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:17.757569 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.757361 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c91a35388f9dd8d39adce6d4d82b0bac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"c91a35388f9dd8d39adce6d4d82b0bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:17.858166 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.858126 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c91a35388f9dd8d39adce6d4d82b0bac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"c91a35388f9dd8d39adce6d4d82b0bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:17.858166 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.858166 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c91a35388f9dd8d39adce6d4d82b0bac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"c91a35388f9dd8d39adce6d4d82b0bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:17.858376 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.858183 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8332e3a50281a723271a56acbbdc782f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"8332e3a50281a723271a56acbbdc782f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:17.858376 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.858232 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c91a35388f9dd8d39adce6d4d82b0bac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"c91a35388f9dd8d39adce6d4d82b0bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:17.858376 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.858245 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c91a35388f9dd8d39adce6d4d82b0bac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"c91a35388f9dd8d39adce6d4d82b0bac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:17.858376 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:17.858275 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8332e3a50281a723271a56acbbdc782f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal\" (UID: \"8332e3a50281a723271a56acbbdc782f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:18.255118 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.255074 5871 apiserver.go:52] "Watching apiserver" Nov 02 22:06:18.370852 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.370812 5871 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 22:06:18.371558 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.371529 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zqhq8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-xfs82","openshift-network-diagnostics/network-check-target-92hpx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd","openshift-image-registry/node-ca-rkr6h","openshift-multus/multus-additional-cni-plugins-4zvx5","openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w","openshift-console/console-6967dc6984-c2sqx","openshift-network-operator/iptables-alerter-mnctf","openshift-console/console-59b677bf6c-t8dfh","openshift-monitoring/node-exporter-gg8nx","openshift-insights/insights-runtime-extractor-tnqdp","openshift-multus/multus-sz594","openshift-ovn-kubernetes/ovnkube-node-hnk92","openshift-dns/node-resolver-vxbxf","openshift-monitoring/metrics-server-7bff846db6-xs49b","openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs","openshift-monitoring/prometheus-k8s-0","kube-system/global-pull-secret-syncer-tp2df","openshift-monitoring/alertmanager-main-0","kube-system/konnectivity-agent-t2vm8","openshift-cluster-node-tuning-operator/tuned-f94lx","openshift-ingress-canary/ingress-canary-lnqpz","kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal","openshift-console/console-75d4dd99d4-vfls2"] Nov 02 22:06:18.381271 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.381242 5871 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:18.381442 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.381276 5871 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:18.422320 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.422286 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 22:06:18.422320 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.422317 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 22:06:18.422320 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.422287 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 22:06:18.422640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.422286 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 22:06:18.422640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.422291 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 22:06:18.422640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.422386 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 22:06:18.422640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.422446 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 22:06:18.422640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.422472 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 22:06:18.422640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.422489 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 22:06:18.422640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.422526 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 22:06:18.422640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.422530 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zcntw\"" Nov 02 22:06:18.422640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.422535 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l4knj\"" Nov 02 22:06:18.422640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.422561 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 22:06:18.422640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.422647 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 22:06:18.423294 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.422651 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 22:06:18.423294 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.422892 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 22:06:18.423294 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.423123 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 22:06:18.423294 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.423142 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 22:06:18.423294 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.423254 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rksbh\"" Nov 02 22:06:18.423294 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.423267 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rsfpb\"" Nov 02 22:06:18.423483 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.423338 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g764w\"" Nov 02 22:06:18.423483 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.423346 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4rxp8\"" Nov 02 22:06:18.425777 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.425756 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-klqgw\"" Nov 02 22:06:18.426042 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426027 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 22:06:18.426224 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426209 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 22:06:18.426307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426282 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 22:06:18.426358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426340 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 22:06:18.426409 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426396 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fcg7g\"" Nov 02 22:06:18.426479 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426463 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 22:06:18.426545 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426514 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 22:06:18.426545 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426536 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:06:18.426610 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426557 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 22:06:18.426682 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426655 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 22:06:18.426682 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426674 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rrhfb\"" Nov 02 22:06:18.426682 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426675 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 22:06:18.426905 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426713 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:06:18.426905 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426716 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 22:06:18.426905 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426716 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 22:06:18.426905 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426854 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-656w2\"" Nov 02 22:06:18.427070 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.426929 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4ts5p\"" Nov 02 22:06:18.427259 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.427245 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 22:06:18.427324 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.427313 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 22:06:18.427366 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.427329 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6jxx8\"" Nov 02 22:06:18.427622 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.427607 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 22:06:18.427743 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.427729 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 22:06:18.427882 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.427857 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 22:06:18.465307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.465271 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 22:06:18.465453 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.465306 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 22:06:18.465453 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.465373 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 22:06:18.465570 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.465458 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 22:06:18.465619 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.465587 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 22:06:18.465619 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.465597 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 22:06:18.465619 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.465590 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 22:06:18.465723 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.465635 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 22:06:18.465930 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.465915 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 22:06:18.465984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.465958 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 22:06:18.465984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.465973 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 22:06:18.466072 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.466059 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 22:06:18.466361 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.466348 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 22:06:18.466438 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.466418 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 22:06:18.466647 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.466630 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 22:06:18.466774 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.466660 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 22:06:18.466774 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.466699 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 22:06:18.466774 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.466699 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 22:06:18.467025 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.466974 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 22:06:18.467532 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.467425 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 22:06:18.467532 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.467447 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9shnq\"" Nov 02 22:06:18.467713 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.467695 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v5dnx\"" Nov 02 22:06:18.468244 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.467943 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4gtmj\"" Nov 02 22:06:18.468244 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.468059 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 22:06:18.468619 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.468595 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 22:06:18.468864 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.468842 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 22:06:18.469133 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.469111 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-88l269il0r23v\"" Nov 02 22:06:18.469824 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.469802 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 22:06:18.470041 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.470022 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q5sg6\"" Nov 02 22:06:18.470240 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.470222 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t4sw7\"" Nov 02 22:06:18.470644 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.470624 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vgqck\"" Nov 02 22:06:18.470718 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.470688 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dmih5pep9rcoc\"" Nov 02 22:06:18.470823 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.470807 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 22:06:18.471143 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.471125 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 22:06:18.471239 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.471180 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cmpr7\"" Nov 02 22:06:18.471239 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.471127 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 22:06:18.471239 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.471215 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hz4mv\"" Nov 02 22:06:18.471467 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.471373 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 22:06:18.472284 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.472228 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 22:06:18.472284 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.472241 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 22:06:18.472505 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.472256 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 22:06:18.472505 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.472278 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 22:06:18.472505 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.472311 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 22:06:18.472505 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.472353 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 22:06:18.472505 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.472384 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 22:06:18.472780 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.472734 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 22:06:18.472780 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.472744 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 22:06:18.472780 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.472766 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 22:06:18.473352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.473331 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 22:06:18.473425 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.473357 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 22:06:18.473425 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.473387 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 22:06:18.473425 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.473412 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 22:06:18.473585 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.473464 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 22:06:18.473585 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.473570 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xhjpw\"" Nov 02 22:06:18.473715 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.473598 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 22:06:18.473715 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.473616 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6jjrn\"" Nov 02 22:06:18.473715 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.473626 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 22:06:18.473715 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.473631 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 22:06:18.474036 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.474015 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 22:06:18.474233 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.474221 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 22:06:18.474845 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.474831 5871 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 22:06:18.513766 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.513667 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 22:06:18.513766 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.513691 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 22:06:18.513995 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.513790 5871 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:06:18.513995 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.513796 5871 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 22:06:18.513995 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:18.513867 5871 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:18.513995 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:18.513915 5871 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-221.us-west-2.compute.internal" Nov 02 22:06:18.514253 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.514234 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 22:06:18.514297 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.514248 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 22:06:18.562797 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.562757 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.562797 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.562797 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-cni-bin\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.563021 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.562817 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:06:18.563021 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.562858 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-client-ca-bundle\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:18.563021 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.562944 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-oauth-serving-cert\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:18.563021 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.562974 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbw9d\" (UniqueName: \"kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-kube-api-access-dbw9d\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.563021 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.562998 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67edaf75-8924-48b9-9a81-93089ab7c82d-metrics-tls\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:06:18.563262 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563026 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-kubernetes\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.563262 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563050 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-cnibin\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.563262 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563075 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.563262 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563102 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-var-lib-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.563262 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563130 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.563262 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563135 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.563262 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563135 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:06:18.563262 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563161 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba6a658f-a491-458f-a3be-2d71c676c069-crio-socket\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:06:18.563262 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563195 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-oauth-serving-cert\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:18.563262 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563207 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-client-ca-bundle\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:18.563262 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563263 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563306 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67edaf75-8924-48b9-9a81-93089ab7c82d-metrics-tls\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563318 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b44243f9-55cd-40b4-87e8-53913b4c26c4-host-slash\") pod \"iptables-alerter-mnctf\" (UID: \"b44243f9-55cd-40b4-87e8-53913b4c26c4\") " pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563366 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563373 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7b199a04-e491-43c9-baf3-0d9e6c663dfa-metrics-server-audit-profiles\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563406 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-accelerators-collector-config\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563433 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-config\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563462 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-sys-fs\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563490 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba6a658f-a491-458f-a3be-2d71c676c069-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563515 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67edaf75-8924-48b9-9a81-93089ab7c82d-config-volume\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563464 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563547 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-system-cni-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563614 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-config\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563629 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-accelerators-collector-config\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563635 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-wtmp\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563683 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba6a658f-a491-458f-a3be-2d71c676c069-data-volume\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563696 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7b199a04-e491-43c9-baf3-0d9e6c663dfa-metrics-server-audit-profiles\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563713 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563735 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba6a658f-a491-458f-a3be-2d71c676c069-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563744 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67edaf75-8924-48b9-9a81-93089ab7c82d-config-volume\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563758 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-oauth-config\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563765 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba6a658f-a491-458f-a3be-2d71c676c069-data-volume\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:06:18.563769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563776 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563802 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563831 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-trusted-ca-bundle\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563892 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-root\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563923 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563950 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/425cd6ec-68c1-48ca-8243-b15082ae9553-konnectivity-ca\") pod \"konnectivity-agent-t2vm8\" (UID: \"425cd6ec-68c1-48ca-8243-b15082ae9553\") " pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563942 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563976 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-oauth-config\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.563994 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf6jx\" (UniqueName: \"kubernetes.io/projected/d725e391-7af5-4691-b327-259e86639f8e-kube-api-access-xf6jx\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564025 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-cni-multus\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564058 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564088 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-secret-metrics-server-tls\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564096 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564113 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b199a04-e491-43c9-baf3-0d9e6c663dfa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564143 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-etc-selinux\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564148 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/425cd6ec-68c1-48ca-8243-b15082ae9553-konnectivity-ca\") pod \"konnectivity-agent-t2vm8\" (UID: \"425cd6ec-68c1-48ca-8243-b15082ae9553\") " pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564168 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/644947bd-1d54-4a7f-afb6-463500cf17b9-kubelet-config\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564174 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-trusted-ca-bundle\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564214 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67edaf75-8924-48b9-9a81-93089ab7c82d-tmp-dir\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564235 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564257 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564309 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67edaf75-8924-48b9-9a81-93089ab7c82d-tmp-dir\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564312 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564332 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-secret-metrics-server-tls\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564353 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-sys\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564356 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b199a04-e491-43c9-baf3-0d9e6c663dfa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564415 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-cni-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564450 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-kubelet\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564479 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-serving-cert\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564483 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564504 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564527 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564545 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbnh2\" (UniqueName: \"kubernetes.io/projected/e95934d1-f5df-4c06-ab39-5e7c8d58858b-kube-api-access-gbnh2\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564560 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:06:18.564575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564578 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564602 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2ph7\" (UniqueName: \"kubernetes.io/projected/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-api-access-c2ph7\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564628 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwv8\" (UniqueName: \"kubernetes.io/projected/ec8310e5-d852-44c1-901b-b54a6c13a798-kube-api-access-5gwv8\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564654 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovn-node-metrics-cert\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564666 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-serving-cert\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564660 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564667 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564678 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovnkube-script-lib\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564718 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nbpz\" (UniqueName: \"kubernetes.io/projected/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-kube-api-access-8nbpz\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564764 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-trusted-ca-bundle\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564792 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564822 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8aa26f0e-38c8-4705-b64a-0a943508b307-host\") pod \"node-ca-rkr6h\" (UID: \"8aa26f0e-38c8-4705-b64a-0a943508b307\") " pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564841 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564847 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovnkube-script-lib\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564927 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovn-node-metrics-cert\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564941 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/644947bd-1d54-4a7f-afb6-463500cf17b9-original-pull-secret\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.564990 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-etc-kubernetes\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565017 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-sys\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565052 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565067 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565083 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b44243f9-55cd-40b4-87e8-53913b4c26c4-iptables-alerter-script\") pod \"iptables-alerter-mnctf\" (UID: \"b44243f9-55cd-40b4-87e8-53913b4c26c4\") " pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565159 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-trusted-ca-bundle\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565172 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba6a658f-a491-458f-a3be-2d71c676c069-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565217 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-system-cni-dir\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565247 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec8310e5-d852-44c1-901b-b54a6c13a798-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565273 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565301 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565342 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565375 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-run-netns\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565349 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba6a658f-a491-458f-a3be-2d71c676c069-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565344 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b44243f9-55cd-40b4-87e8-53913b4c26c4-iptables-alerter-script\") pod \"iptables-alerter-mnctf\" (UID: \"b44243f9-55cd-40b4-87e8-53913b4c26c4\") " pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565442 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565482 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysctl-conf\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565501 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec8310e5-d852-44c1-901b-b54a6c13a798-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565509 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d725e391-7af5-4691-b327-259e86639f8e-etc-tuned\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565522 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565539 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwll\" (UniqueName: \"kubernetes.io/projected/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-kube-api-access-qzwll\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565521 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565567 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-secret-metrics-server-client-certs\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565585 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d725e391-7af5-4691-b327-259e86639f8e-etc-tuned\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565590 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-run\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565607 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-tls\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565714 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-etc-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565745 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-log-socket\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565772 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/425cd6ec-68c1-48ca-8243-b15082ae9553-agent-certs\") pod \"konnectivity-agent-t2vm8\" (UID: \"425cd6ec-68c1-48ca-8243-b15082ae9553\") " pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565798 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-k8s-cni-cncf-io\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565803 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7b199a04-e491-43c9-baf3-0d9e6c663dfa-secret-metrics-server-client-certs\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565837 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e95934d1-f5df-4c06-ab39-5e7c8d58858b-tmp\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565918 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-serving-cert\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565920 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-tls\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565957 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/759f8e6e-92ee-4b83-9447-949cb34a2cd1-metrics-client-ca\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.565984 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566009 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-cni-netd\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566037 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-socket-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566066 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9b2t\" (UniqueName: \"kubernetes.io/projected/ba6a658f-a491-458f-a3be-2d71c676c069-kube-api-access-w9b2t\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566071 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/425cd6ec-68c1-48ca-8243-b15082ae9553-agent-certs\") pod \"konnectivity-agent-t2vm8\" (UID: \"425cd6ec-68c1-48ca-8243-b15082ae9553\") " pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566091 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-config\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566100 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e95934d1-f5df-4c06-ab39-5e7c8d58858b-tmp\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566136 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-serving-cert\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:18.566117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566135 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz6cr\" (UniqueName: \"kubernetes.io/projected/be31cb75-89a7-4b84-b26f-2050acfa249c-kube-api-access-bz6cr\") pod \"node-resolver-vxbxf\" (UID: \"be31cb75-89a7-4b84-b26f-2050acfa249c\") " pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566174 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/759f8e6e-92ee-4b83-9447-949cb34a2cd1-metrics-client-ca\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566179 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xbbh\" (UniqueName: \"kubernetes.io/projected/759f8e6e-92ee-4b83-9447-949cb34a2cd1-kube-api-access-4xbbh\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566208 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566236 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566261 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-config-out\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566311 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-cni-bin\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566333 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovnkube-config\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566356 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsphb\" (UniqueName: \"kubernetes.io/projected/2fb97b10-9112-4dd4-8d7d-a2c864c39358-kube-api-access-bsphb\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566381 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cni-binary-copy\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566405 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-textfile\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566408 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566427 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-ovn\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566438 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566452 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-cni-binary-copy\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566483 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-oauth-config\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566511 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2xh\" (UniqueName: \"kubernetes.io/projected/7d6c8f46-7560-45fb-8d07-f32136e14bce-kube-api-access-5p2xh\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566525 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-config-out\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566537 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sql9\" (UniqueName: \"kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-kube-api-access-4sql9\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566565 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-systemd\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566592 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysconfig\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566625 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-oauth-serving-cert\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566628 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-ovnkube-config\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566663 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-tmp\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566704 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-web-config\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566726 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-textfile\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566735 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn52n\" (UniqueName: \"kubernetes.io/projected/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-kube-api-access-pn52n\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566766 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-kubelet\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566792 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8aa26f0e-38c8-4705-b64a-0a943508b307-serviceca\") pod \"node-ca-rkr6h\" (UID: \"8aa26f0e-38c8-4705-b64a-0a943508b307\") " pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566798 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-tmp\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566804 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-config\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566819 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566819 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7b199a04-e491-43c9-baf3-0d9e6c663dfa-audit-log\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566852 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-cni-binary-copy\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.566928 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7b199a04-e491-43c9-baf3-0d9e6c663dfa-audit-log\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.567236 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cni-binary-copy\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.567399 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-oauth-serving-cert\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.567448 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-oauth-config\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.567474 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8aa26f0e-38c8-4705-b64a-0a943508b307-serviceca\") pod \"node-ca-rkr6h\" (UID: \"8aa26f0e-38c8-4705-b64a-0a943508b307\") " pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.567591 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-web-config\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.567594 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-serving-cert\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.567693 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-config\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.567746 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-service-ca\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.567812 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-serving-cert\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.567817 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.567863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.567912 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.567939 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.567964 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npbhr\" (UniqueName: \"kubernetes.io/projected/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-kube-api-access-npbhr\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.567991 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-service-ca\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.567994 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-lib-modules\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568031 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-socket-dir-parent\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568063 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-hostroot\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568062 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568092 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjx2f\" (UniqueName: \"kubernetes.io/projected/7b199a04-e491-43c9-baf3-0d9e6c663dfa-kube-api-access-vjx2f\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568111 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568139 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568168 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-systemd-units\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568230 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70d481aa-8880-41b2-957a-285385bceebd-cert\") pod \"ingress-canary-lnqpz\" (UID: \"70d481aa-8880-41b2-957a-285385bceebd\") " pod="openshift-ingress-canary/ingress-canary-lnqpz" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568261 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec8310e5-d852-44c1-901b-b54a6c13a798-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568290 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec8310e5-d852-44c1-901b-b54a6c13a798-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568314 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-config\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568319 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-web-config\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568373 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-netns\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568402 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzqmd\" (UniqueName: \"kubernetes.io/projected/70d481aa-8880-41b2-957a-285385bceebd-kube-api-access-bzqmd\") pod \"ingress-canary-lnqpz\" (UID: \"70d481aa-8880-41b2-957a-285385bceebd\") " pod="openshift-ingress-canary/ingress-canary-lnqpz" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568431 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-oauth-serving-cert\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568456 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d6c8f46-7560-45fb-8d07-f32136e14bce-tmp\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568454 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568483 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568497 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-web-config\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568508 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysctl-d\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568521 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70d481aa-8880-41b2-957a-285385bceebd-cert\") pod \"ingress-canary-lnqpz\" (UID: \"70d481aa-8880-41b2-957a-285385bceebd\") " pod="openshift-ingress-canary/ingress-canary-lnqpz" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568523 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec8310e5-d852-44c1-901b-b54a6c13a798-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568651 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec8310e5-d852-44c1-901b-b54a6c13a798-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568670 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-conf-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568710 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-oauth-serving-cert\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568723 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-daemon-config\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568769 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d6c8f46-7560-45fb-8d07-f32136e14bce-tmp\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568794 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568801 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-multus-certs\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568839 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r6hz\" (UniqueName: \"kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz\") pod \"network-check-target-92hpx\" (UID: \"670c59c6-f81f-47b5-9624-2bab9d1fc847\") " pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568900 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.568951 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-daemon-config\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569014 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxq4\" (UniqueName: \"kubernetes.io/projected/67edaf75-8924-48b9-9a81-93089ab7c82d-kube-api-access-cfxq4\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569074 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-modprobe-d\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569112 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569156 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569170 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-host\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569219 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-service-ca\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569277 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569334 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569421 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-service-ca\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569423 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-registration-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569440 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569538 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-node-log\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569584 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwzb\" (UniqueName: \"kubernetes.io/projected/b44243f9-55cd-40b4-87e8-53913b4c26c4-kube-api-access-qgwzb\") pod \"iptables-alerter-mnctf\" (UID: \"b44243f9-55cd-40b4-87e8-53913b4c26c4\") " pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569617 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-systemd\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569619 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569679 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569704 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569720 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569737 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569752 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-var-lib-kubelet\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569768 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d725e391-7af5-4691-b327-259e86639f8e-tmp\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569898 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569907 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-config\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569955 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzj5\" (UniqueName: \"kubernetes.io/projected/6ae0899d-36a2-45b1-9bdf-a76acc087491-kube-api-access-2jzj5\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.569984 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be31cb75-89a7-4b84-b26f-2050acfa249c-tmp-dir\") pod \"node-resolver-vxbxf\" (UID: \"be31cb75-89a7-4b84-b26f-2050acfa249c\") " pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:06:18.569975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570010 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-service-ca\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570033 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-config\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570033 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570040 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d725e391-7af5-4691-b327-259e86639f8e-tmp\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570067 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570082 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae0899d-36a2-45b1-9bdf-a76acc087491-metrics-certs\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570095 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-slash\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570109 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be31cb75-89a7-4b84-b26f-2050acfa249c-tmp-dir\") pod \"node-resolver-vxbxf\" (UID: \"be31cb75-89a7-4b84-b26f-2050acfa249c\") " pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570186 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570203 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570206 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-service-ca\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570279 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570289 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-env-overrides\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570355 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/644947bd-1d54-4a7f-afb6-463500cf17b9-dbus\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570377 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cnibin\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570390 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570395 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-out\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570398 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2fb97b10-9112-4dd4-8d7d-a2c864c39358-env-overrides\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570426 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570456 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-device-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570466 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-out\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570505 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6g87\" (UniqueName: \"kubernetes.io/projected/8aa26f0e-38c8-4705-b64a-0a943508b307-kube-api-access-m6g87\") pod \"node-ca-rkr6h\" (UID: \"8aa26f0e-38c8-4705-b64a-0a943508b307\") " pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570530 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be31cb75-89a7-4b84-b26f-2050acfa249c-hosts-file\") pod \"node-resolver-vxbxf\" (UID: \"be31cb75-89a7-4b84-b26f-2050acfa249c\") " pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570580 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-oauth-config\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570608 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570611 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570636 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570751 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-oauth-config\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570778 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.571794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.570803 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.619141 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.619098 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf6jx\" (UniqueName: \"kubernetes.io/projected/d725e391-7af5-4691-b327-259e86639f8e-kube-api-access-xf6jx\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.619374 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.619347 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbw9d\" (UniqueName: \"kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-kube-api-access-dbw9d\") pod \"prometheus-k8s-0\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:18.619683 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.619662 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gwv8\" (UniqueName: \"kubernetes.io/projected/ec8310e5-d852-44c1-901b-b54a6c13a798-kube-api-access-5gwv8\") pod \"openshift-state-metrics-7bd7489957-xzrgs\" (UID: \"ec8310e5-d852-44c1-901b-b54a6c13a798\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xzrgs" Nov 02 22:06:18.619908 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.619887 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbnh2\" (UniqueName: \"kubernetes.io/projected/e95934d1-f5df-4c06-ab39-5e7c8d58858b-kube-api-access-gbnh2\") pod \"console-59b677bf6c-t8dfh\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:18.620098 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.620082 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwll\" (UniqueName: \"kubernetes.io/projected/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-kube-api-access-qzwll\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.620207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.620191 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nbpz\" (UniqueName: \"kubernetes.io/projected/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-kube-api-access-8nbpz\") pod \"console-6967dc6984-c2sqx\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:18.620296 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.620281 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2ph7\" (UniqueName: \"kubernetes.io/projected/1f3ab9d6-1afa-499c-80d2-a1d11981eeb2-kube-api-access-c2ph7\") pod \"kube-state-metrics-5dc8d7d89b-g729w\" (UID: \"1f3ab9d6-1afa-499c-80d2-a1d11981eeb2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g729w" Nov 02 22:06:18.621438 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.621417 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9b2t\" (UniqueName: \"kubernetes.io/projected/ba6a658f-a491-458f-a3be-2d71c676c069-kube-api-access-w9b2t\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:06:18.621541 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.621512 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xbbh\" (UniqueName: \"kubernetes.io/projected/759f8e6e-92ee-4b83-9447-949cb34a2cd1-kube-api-access-4xbbh\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.622255 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.622237 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsphb\" (UniqueName: \"kubernetes.io/projected/2fb97b10-9112-4dd4-8d7d-a2c864c39358-kube-api-access-bsphb\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.622461 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.622445 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz6cr\" (UniqueName: \"kubernetes.io/projected/be31cb75-89a7-4b84-b26f-2050acfa249c-kube-api-access-bz6cr\") pod \"node-resolver-vxbxf\" (UID: \"be31cb75-89a7-4b84-b26f-2050acfa249c\") " pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:06:18.622763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.622748 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn52n\" (UniqueName: \"kubernetes.io/projected/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-kube-api-access-pn52n\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.623943 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.623927 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2xh\" (UniqueName: \"kubernetes.io/projected/7d6c8f46-7560-45fb-8d07-f32136e14bce-kube-api-access-5p2xh\") pod \"console-75d4dd99d4-vfls2\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:18.624197 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.624183 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjx2f\" (UniqueName: \"kubernetes.io/projected/7b199a04-e491-43c9-baf3-0d9e6c663dfa-kube-api-access-vjx2f\") pod \"metrics-server-7bff846db6-xs49b\" (UID: \"7b199a04-e491-43c9-baf3-0d9e6c663dfa\") " pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:18.626016 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.625987 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfxq4\" (UniqueName: \"kubernetes.io/projected/67edaf75-8924-48b9-9a81-93089ab7c82d-kube-api-access-cfxq4\") pod \"dns-default-zqhq8\" (UID: \"67edaf75-8924-48b9-9a81-93089ab7c82d\") " pod="openshift-dns/dns-default-zqhq8" Nov 02 22:06:18.626109 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.626076 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jzj5\" (UniqueName: \"kubernetes.io/projected/6ae0899d-36a2-45b1-9bdf-a76acc087491-kube-api-access-2jzj5\") pod \"network-metrics-daemon-xfs82\" (UID: \"6ae0899d-36a2-45b1-9bdf-a76acc087491\") " pod="openshift-multus/network-metrics-daemon-xfs82" Nov 02 22:06:18.626275 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.626258 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwzb\" (UniqueName: \"kubernetes.io/projected/b44243f9-55cd-40b4-87e8-53913b4c26c4-kube-api-access-qgwzb\") pod \"iptables-alerter-mnctf\" (UID: \"b44243f9-55cd-40b4-87e8-53913b4c26c4\") " pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:06:18.626340 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.626279 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npbhr\" (UniqueName: \"kubernetes.io/projected/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-kube-api-access-npbhr\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.626405 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.626385 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r6hz\" (UniqueName: \"kubernetes.io/projected/670c59c6-f81f-47b5-9624-2bab9d1fc847-kube-api-access-4r6hz\") pod \"network-check-target-92hpx\" (UID: \"670c59c6-f81f-47b5-9624-2bab9d1fc847\") " pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:06:18.626535 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.626521 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzqmd\" (UniqueName: \"kubernetes.io/projected/70d481aa-8880-41b2-957a-285385bceebd-kube-api-access-bzqmd\") pod \"ingress-canary-lnqpz\" (UID: \"70d481aa-8880-41b2-957a-285385bceebd\") " pod="openshift-ingress-canary/ingress-canary-lnqpz" Nov 02 22:06:18.629182 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.629160 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sql9\" (UniqueName: \"kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-kube-api-access-4sql9\") pod \"alertmanager-main-0\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:18.631782 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.631765 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6g87\" (UniqueName: \"kubernetes.io/projected/8aa26f0e-38c8-4705-b64a-0a943508b307-kube-api-access-m6g87\") pod \"node-ca-rkr6h\" (UID: \"8aa26f0e-38c8-4705-b64a-0a943508b307\") " pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:06:18.671204 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671179 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.671313 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671211 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8aa26f0e-38c8-4705-b64a-0a943508b307-host\") pod \"node-ca-rkr6h\" (UID: \"8aa26f0e-38c8-4705-b64a-0a943508b307\") " pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:06:18.671313 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671231 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-etc-kubernetes\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.671313 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671266 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-sys\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.671313 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671289 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.671313 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671297 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.671313 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671301 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-etc-kubernetes\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.671534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671305 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8aa26f0e-38c8-4705-b64a-0a943508b307-host\") pod \"node-ca-rkr6h\" (UID: \"8aa26f0e-38c8-4705-b64a-0a943508b307\") " pod="openshift-image-registry/node-ca-rkr6h" Nov 02 22:06:18.671534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671333 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-sys\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.671534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671346 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-system-cni-dir\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.671534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671367 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.671534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671379 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-run-netns\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.671534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671389 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-system-cni-dir\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.671534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671405 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysctl-conf\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.671534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671430 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-run\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.671534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671442 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-run-netns\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.671534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671451 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-etc-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.671534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671487 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-etc-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.671534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671492 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-run\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.671534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671515 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-log-socket\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.671534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671524 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysctl-conf\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.671534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671528 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-log-socket\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.671534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671534 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-k8s-cni-cncf-io\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671563 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-cni-netd\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671579 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-socket-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671589 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-k8s-cni-cncf-io\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671598 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-cni-bin\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671614 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-ovn\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671616 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-cni-netd\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671630 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-socket-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671638 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-systemd\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671655 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysconfig\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671667 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-ovn\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671673 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-systemd\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671674 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-kubelet\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671655 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-cni-bin\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671704 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-lib-modules\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671713 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysconfig\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671717 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-kubelet\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671755 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-socket-dir-parent\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671780 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-hostroot\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671813 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-systemd-units\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671841 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-netns\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671849 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-lib-modules\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671851 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-hostroot\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671868 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysctl-d\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671893 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-socket-dir-parent\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671906 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-netns\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671903 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-systemd-units\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671908 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-conf-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671934 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-conf-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671946 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-multus-certs\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671970 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-modprobe-d\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671981 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-sysctl-d\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671986 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-host\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.671993 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-run-multus-certs\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672007 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-registration-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672011 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-host\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672031 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-node-log\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672053 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-registration-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672055 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-modprobe-d\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672053 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-systemd\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672080 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-node-log\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672085 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672101 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-var-lib-kubelet\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672112 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672120 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-slash\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672135 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/644947bd-1d54-4a7f-afb6-463500cf17b9-dbus\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:06:18.672124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672150 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cnibin\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672166 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-device-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672086 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-systemd\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672182 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-var-lib-kubelet\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672188 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be31cb75-89a7-4b84-b26f-2050acfa249c-hosts-file\") pod \"node-resolver-vxbxf\" (UID: \"be31cb75-89a7-4b84-b26f-2050acfa249c\") " pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672194 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3-cnibin\") pod \"multus-additional-cni-plugins-4zvx5\" (UID: \"44e3a7cc-c4c9-44d7-b358-029b6a8c9cc3\") " pod="openshift-multus/multus-additional-cni-plugins-4zvx5" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672184 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-slash\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672222 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-device-dir\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672223 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/644947bd-1d54-4a7f-afb6-463500cf17b9-dbus\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672241 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be31cb75-89a7-4b84-b26f-2050acfa249c-hosts-file\") pod \"node-resolver-vxbxf\" (UID: \"be31cb75-89a7-4b84-b26f-2050acfa249c\") " pod="openshift-dns/node-resolver-vxbxf" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672278 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-cni-bin\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672311 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-cni-bin\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672361 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-kubernetes\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672315 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-etc-kubernetes\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672406 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-cnibin\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672434 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-var-lib-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672451 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-cnibin\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672460 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672485 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba6a658f-a491-458f-a3be-2d71c676c069-crio-socket\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672492 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672486 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-var-lib-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672512 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba6a658f-a491-458f-a3be-2d71c676c069-crio-socket\") pod \"insights-runtime-extractor-tnqdp\" (UID: \"ba6a658f-a491-458f-a3be-2d71c676c069\") " pod="openshift-insights/insights-runtime-extractor-tnqdp" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672511 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b44243f9-55cd-40b4-87e8-53913b4c26c4-host-slash\") pod \"iptables-alerter-mnctf\" (UID: \"b44243f9-55cd-40b4-87e8-53913b4c26c4\") " pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672537 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b44243f9-55cd-40b4-87e8-53913b4c26c4-host-slash\") pod \"iptables-alerter-mnctf\" (UID: \"b44243f9-55cd-40b4-87e8-53913b4c26c4\") " pod="openshift-network-operator/iptables-alerter-mnctf" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672539 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-sys-fs\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672563 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-sys-fs\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672661 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-system-cni-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672689 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-wtmp\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672719 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672741 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-root\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672751 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-system-cni-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672759 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-node-exporter-wtmp\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672766 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2fb97b10-9112-4dd4-8d7d-a2c864c39358-run-openvswitch\") pod \"ovnkube-node-hnk92\" (UID: \"2fb97b10-9112-4dd4-8d7d-a2c864c39358\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672768 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-cni-multus\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672784 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/759f8e6e-92ee-4b83-9447-949cb34a2cd1-root\") pod \"node-exporter-gg8nx\" (UID: \"759f8e6e-92ee-4b83-9447-949cb34a2cd1\") " pod="openshift-monitoring/node-exporter-gg8nx" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672800 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-cni-multus\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672802 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-etc-selinux\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672830 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/644947bd-1d54-4a7f-afb6-463500cf17b9-kubelet-config\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672839 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8b8a9a43-5fd2-4e49-b313-9ff58be7524b-etc-selinux\") pod \"aws-ebs-csi-driver-node-kpsgd\" (UID: \"8b8a9a43-5fd2-4e49-b313-9ff58be7524b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpsgd" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672857 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-sys\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672900 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-cni-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672905 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/644947bd-1d54-4a7f-afb6-463500cf17b9-kubelet-config\") pod \"global-pull-secret-syncer-tp2df\" (UID: \"644947bd-1d54-4a7f-afb6-463500cf17b9\") " pod="kube-system/global-pull-secret-syncer-tp2df" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672925 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-kubelet\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672940 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d725e391-7af5-4691-b327-259e86639f8e-sys\") pod \"tuned-f94lx\" (UID: \"d725e391-7af5-4691-b327-259e86639f8e\") " pod="openshift-cluster-node-tuning-operator/tuned-f94lx" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.672956 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-multus-cni-dir\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.673678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.673014 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f6652ca-37d9-45f8-a135-19dc4cc60fc3-host-var-lib-kubelet\") pod \"multus-sz594\" (UID: \"2f6652ca-37d9-45f8-a135-19dc4cc60fc3\") " pod="openshift-multus/multus-sz594" Nov 02 22:06:18.811730 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:18.811651 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zqhq8" Nov 02 22:06:20.004829 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:20.004791 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:06:20.181981 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:20.181938 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6967dc6984-c2sqx"] Nov 02 22:06:20.526890 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:20.526845 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:22.575382 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:22.575344 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:22.594950 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:22.594923 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:23.080474 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:23.080426 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:23.258889 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:23.258833 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:23.627110 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:23.627075 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:23.627510 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:23.627312 5871 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 22:06:24.832825 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:24.832740 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:24.836939 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:24.836918 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:25.544331 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:25.544293 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:25.728334 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:25.728294 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:25.899792 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:25.899754 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:26.555249 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:26.555210 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:27.223216 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:27.223170 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:27.391700 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:27.391661 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:29.257231 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:29.257203 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7bff846db6-xs49b" Nov 02 22:06:30.401957 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:30.401914 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:30.570143 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:30.570102 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:31.124358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.124317 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-cc8bc6b4f-t55f8"] Nov 02 22:06:31.174985 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.174953 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.223129 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.223087 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cc8bc6b4f-t55f8"] Nov 02 22:06:31.277469 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.277429 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36558817-3041-40e9-8335-858adda9c796-oauth-serving-cert\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.277646 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.277475 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sd9l\" (UniqueName: \"kubernetes.io/projected/36558817-3041-40e9-8335-858adda9c796-kube-api-access-2sd9l\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.277646 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.277622 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36558817-3041-40e9-8335-858adda9c796-console-config\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.277723 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.277656 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36558817-3041-40e9-8335-858adda9c796-tmp\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.277756 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.277722 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36558817-3041-40e9-8335-858adda9c796-console-serving-cert\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.277789 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.277770 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36558817-3041-40e9-8335-858adda9c796-console-oauth-config\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.277829 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.277815 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36558817-3041-40e9-8335-858adda9c796-service-ca\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.277862 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.277855 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36558817-3041-40e9-8335-858adda9c796-trusted-ca-bundle\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.379302 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.379215 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36558817-3041-40e9-8335-858adda9c796-console-serving-cert\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.379302 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.379267 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36558817-3041-40e9-8335-858adda9c796-console-oauth-config\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.379302 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.379284 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36558817-3041-40e9-8335-858adda9c796-service-ca\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.379550 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.379354 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36558817-3041-40e9-8335-858adda9c796-trusted-ca-bundle\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.379550 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.379411 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36558817-3041-40e9-8335-858adda9c796-oauth-serving-cert\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.379550 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.379445 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sd9l\" (UniqueName: \"kubernetes.io/projected/36558817-3041-40e9-8335-858adda9c796-kube-api-access-2sd9l\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.379550 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.379496 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36558817-3041-40e9-8335-858adda9c796-console-config\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.379550 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.379513 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36558817-3041-40e9-8335-858adda9c796-tmp\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.379803 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.379670 5871 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 22:06:31.380013 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.379980 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36558817-3041-40e9-8335-858adda9c796-tmp\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.380175 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.380148 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36558817-3041-40e9-8335-858adda9c796-service-ca\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.380332 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.380304 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36558817-3041-40e9-8335-858adda9c796-trusted-ca-bundle\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.380332 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.380226 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36558817-3041-40e9-8335-858adda9c796-oauth-serving-cert\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.380450 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.380285 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36558817-3041-40e9-8335-858adda9c796-console-config\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.381936 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.381917 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36558817-3041-40e9-8335-858adda9c796-console-serving-cert\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.382152 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.382136 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36558817-3041-40e9-8335-858adda9c796-console-oauth-config\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.437705 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.437677 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sd9l\" (UniqueName: \"kubernetes.io/projected/36558817-3041-40e9-8335-858adda9c796-kube-api-access-2sd9l\") pod \"console-cc8bc6b4f-t55f8\" (UID: \"36558817-3041-40e9-8335-858adda9c796\") " pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.484758 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.484720 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:31.495737 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.495698 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:31.671555 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.671476 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:31.894348 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:31.894305 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cc8bc6b4f-t55f8"] Nov 02 22:06:31.902434 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:31.902402 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36558817_3041_40e9_8335_858adda9c796.slice/crio-1709c78acb263e1f79a98589ed06abc795c32f0d72da5fe4a81535e1282fadf0 WatchSource:0}: Error finding container 1709c78acb263e1f79a98589ed06abc795c32f0d72da5fe4a81535e1282fadf0: Status 404 returned error can't find the container with id 1709c78acb263e1f79a98589ed06abc795c32f0d72da5fe4a81535e1282fadf0 Nov 02 22:06:32.438580 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:32.438539 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc8bc6b4f-t55f8" event={"ID":"36558817-3041-40e9-8335-858adda9c796","Type":"ContainerStarted","Data":"60156651a58c38cc9bb45dde510c8508c9e9b4a0017971cd9e11d9aaea3cfe0e"} Nov 02 22:06:32.439107 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:32.438603 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc8bc6b4f-t55f8" event={"ID":"36558817-3041-40e9-8335-858adda9c796","Type":"ContainerStarted","Data":"1709c78acb263e1f79a98589ed06abc795c32f0d72da5fe4a81535e1282fadf0"} Nov 02 22:06:32.550061 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:32.549949 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-cc8bc6b4f-t55f8" podStartSLOduration=1.549934295 podStartE2EDuration="1.549934295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:32.549515868 +0000 UTC m=+15.361846046" watchObservedRunningTime="2026-11-02 22:06:32.549934295 +0000 UTC m=+15.362264470" Nov 02 22:06:32.807708 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:32.807624 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:32.986790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:32.986757 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:34.976318 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:34.976284 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:35.152153 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:35.152125 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:35.904959 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:35.904925 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:06:36.816219 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:36.816181 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:36.993319 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:36.993283 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:37.920275 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:37.920231 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:06:37.920783 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:37.920621 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="alertmanager" containerID="cri-o://b26d63ec309e5fe09be31f4a05a4bea233fd45eca992aa8e51fd2d6e209ed189" gracePeriod=120 Nov 02 22:06:37.920783 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:37.920662 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="kube-rbac-proxy-metric" containerID="cri-o://dc00cab91a4160629f3a6a7df16ffbfa5feaa5d34c1f42c27d0a9f62996b5207" gracePeriod=120 Nov 02 22:06:37.920783 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:37.920705 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="kube-rbac-proxy" containerID="cri-o://b0c2b65626f6d3796317124b61409892a23efe75bb852073d089f2314e02a792" gracePeriod=120 Nov 02 22:06:37.920783 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:37.920699 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="kube-rbac-proxy-web" containerID="cri-o://4c6bfcbe1ed2682d3f93aa1b9f6cebed5f6034ac176331a328f14b458fe56582" gracePeriod=120 Nov 02 22:06:37.920783 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:37.920733 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="prom-label-proxy" containerID="cri-o://2da91375ef30cbd47ccd84f5d23c10081042db722eb215c8af62441e3e2f6684" gracePeriod=120 Nov 02 22:06:37.921144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:37.920735 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="config-reloader" containerID="cri-o://5891915245a856c716b3457d18b77ebc7a28274b31169b43ce4a646d3728ef74" gracePeriod=120 Nov 02 22:06:38.272329 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:38.272247 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnk92" Nov 02 22:06:38.464277 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:38.464242 5871 generic.go:356] "Generic (PLEG): container finished" podID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerID="2da91375ef30cbd47ccd84f5d23c10081042db722eb215c8af62441e3e2f6684" exitCode=0 Nov 02 22:06:38.464277 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:38.464268 5871 generic.go:356] "Generic (PLEG): container finished" podID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerID="b0c2b65626f6d3796317124b61409892a23efe75bb852073d089f2314e02a792" exitCode=0 Nov 02 22:06:38.464277 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:38.464274 5871 generic.go:356] "Generic (PLEG): container finished" podID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerID="4c6bfcbe1ed2682d3f93aa1b9f6cebed5f6034ac176331a328f14b458fe56582" exitCode=0 Nov 02 22:06:38.464277 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:38.464281 5871 generic.go:356] "Generic (PLEG): container finished" podID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerID="5891915245a856c716b3457d18b77ebc7a28274b31169b43ce4a646d3728ef74" exitCode=0 Nov 02 22:06:38.464277 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:38.464286 5871 generic.go:356] "Generic (PLEG): container finished" podID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerID="b26d63ec309e5fe09be31f4a05a4bea233fd45eca992aa8e51fd2d6e209ed189" exitCode=0 Nov 02 22:06:38.464559 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:38.464305 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4","Type":"ContainerDied","Data":"2da91375ef30cbd47ccd84f5d23c10081042db722eb215c8af62441e3e2f6684"} Nov 02 22:06:38.464559 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:38.464373 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4","Type":"ContainerDied","Data":"b0c2b65626f6d3796317124b61409892a23efe75bb852073d089f2314e02a792"} Nov 02 22:06:38.464559 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:38.464401 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4","Type":"ContainerDied","Data":"4c6bfcbe1ed2682d3f93aa1b9f6cebed5f6034ac176331a328f14b458fe56582"} Nov 02 22:06:38.464559 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:38.464424 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4","Type":"ContainerDied","Data":"5891915245a856c716b3457d18b77ebc7a28274b31169b43ce4a646d3728ef74"} Nov 02 22:06:38.464559 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:38.464453 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4","Type":"ContainerDied","Data":"b26d63ec309e5fe09be31f4a05a4bea233fd45eca992aa8e51fd2d6e209ed189"} Nov 02 22:06:38.635059 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:38.635029 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t2vm8" Nov 02 22:06:39.177241 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.177215 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.248545 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.248458 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-volume\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-volume\") pod \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " Nov 02 22:06:39.248545 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.248520 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy\") pod \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " Nov 02 22:06:39.248545 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.248543 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-main-tls\") pod \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " Nov 02 22:06:39.248812 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.248574 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-main-db\") pod \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " Nov 02 22:06:39.248812 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.248596 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " Nov 02 22:06:39.248812 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.248633 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-cluster-tls-config\") pod \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " Nov 02 22:06:39.248812 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.248678 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-kube-api-access-4sql9\" (UniqueName: \"kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-kube-api-access-4sql9\") pod \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " Nov 02 22:06:39.248812 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.248722 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-web\") pod \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " Nov 02 22:06:39.248812 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.248756 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-metrics-client-ca\") pod \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " Nov 02 22:06:39.248812 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.248810 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-web-config\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-web-config\") pod \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " Nov 02 22:06:39.249222 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.248861 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-tls-assets\" (UniqueName: \"kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-tls-assets\") pod \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " Nov 02 22:06:39.249222 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.248928 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-out\") pod \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " Nov 02 22:06:39.249222 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.248974 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-trusted-ca-bundle\") pod \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\" (UID: \"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4\") " Nov 02 22:06:39.249222 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.249016 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-main-db" pod "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" (UID: "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:06:39.249433 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.249269 5871 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-main-db\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.249433 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.249266 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-metrics-client-ca" pod "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" (UID: "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:39.251523 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.251491 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-trusted-ca-bundle" pod "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" (UID: "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:39.251846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.251782 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-metric" pod "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" (UID: "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:39.252237 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.252211 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-main-tls" pod "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" (UID: "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:39.252237 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.252209 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-tls-assets" pod "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" (UID: "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:06:39.252395 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.252322 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-kube-api-access-4sql9" pod "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" (UID: "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4"). InnerVolumeSpecName "kube-api-access-4sql9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:06:39.252395 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.252337 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-volume" pod "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" (UID: "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:39.252474 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.252390 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-web" pod "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" (UID: "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:39.252474 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.252347 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-out" pod "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" (UID: "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:06:39.253207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.253184 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy" pod "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" (UID: "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:39.257522 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.257496 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-cluster-tls-config" pod "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" (UID: "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:39.264220 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.264190 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-web-config" pod "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" (UID: "a0d7bd65-c3af-49fa-9c77-8c7c348e73f4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:39.353960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.350359 5871 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-metrics-client-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.353960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.350553 5871 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-web-config\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.353960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.350573 5871 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-tls-assets\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.353960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.350588 5871 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-out\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.353960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.350614 5871 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.353960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.350630 5871 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-config-volume\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.353960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.350645 5871 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.353960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.350661 5871 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-main-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.353960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.350681 5871 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.353960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.350696 5871 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-cluster-tls-config\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.353960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.350711 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sql9\" (UniqueName: \"kubernetes.io/projected/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-kube-api-access-4sql9\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.353960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.350725 5871 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:39.471226 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.471185 5871 generic.go:356] "Generic (PLEG): container finished" podID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerID="dc00cab91a4160629f3a6a7df16ffbfa5feaa5d34c1f42c27d0a9f62996b5207" exitCode=0 Nov 02 22:06:39.471395 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.471243 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4","Type":"ContainerDied","Data":"dc00cab91a4160629f3a6a7df16ffbfa5feaa5d34c1f42c27d0a9f62996b5207"} Nov 02 22:06:39.471395 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.471299 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 22:06:39.471395 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.471349 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0d7bd65-c3af-49fa-9c77-8c7c348e73f4","Type":"ContainerDied","Data":"688a334dbdb09643b4eeffbd2800ec19eb51db27b196952cc2361e0226cd28e7"} Nov 02 22:06:39.471395 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.471394 5871 scope.go:122] "RemoveContainer" containerID="2da91375ef30cbd47ccd84f5d23c10081042db722eb215c8af62441e3e2f6684" Nov 02 22:06:39.481663 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.481629 5871 scope.go:122] "RemoveContainer" containerID="dc00cab91a4160629f3a6a7df16ffbfa5feaa5d34c1f42c27d0a9f62996b5207" Nov 02 22:06:39.491366 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.491342 5871 scope.go:122] "RemoveContainer" containerID="b0c2b65626f6d3796317124b61409892a23efe75bb852073d089f2314e02a792" Nov 02 22:06:39.499061 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.499012 5871 scope.go:122] "RemoveContainer" containerID="4c6bfcbe1ed2682d3f93aa1b9f6cebed5f6034ac176331a328f14b458fe56582" Nov 02 22:06:39.508014 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.507960 5871 scope.go:122] "RemoveContainer" containerID="5891915245a856c716b3457d18b77ebc7a28274b31169b43ce4a646d3728ef74" Nov 02 22:06:39.516841 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.516820 5871 scope.go:122] "RemoveContainer" containerID="b26d63ec309e5fe09be31f4a05a4bea233fd45eca992aa8e51fd2d6e209ed189" Nov 02 22:06:39.525433 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.525409 5871 scope.go:122] "RemoveContainer" containerID="5064217c1727c131b0e57008af6162977bb22e38ebaf8db195cb216cf2f4ee4d" Nov 02 22:06:39.533704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.533670 5871 scope.go:122] "RemoveContainer" containerID="2da91375ef30cbd47ccd84f5d23c10081042db722eb215c8af62441e3e2f6684" Nov 02 22:06:39.534115 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:39.534092 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da91375ef30cbd47ccd84f5d23c10081042db722eb215c8af62441e3e2f6684\": container with ID starting with 2da91375ef30cbd47ccd84f5d23c10081042db722eb215c8af62441e3e2f6684 not found: ID does not exist" containerID="2da91375ef30cbd47ccd84f5d23c10081042db722eb215c8af62441e3e2f6684" Nov 02 22:06:39.534196 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.534128 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da91375ef30cbd47ccd84f5d23c10081042db722eb215c8af62441e3e2f6684"} err="failed to get container status \"2da91375ef30cbd47ccd84f5d23c10081042db722eb215c8af62441e3e2f6684\": rpc error: code = NotFound desc = could not find container \"2da91375ef30cbd47ccd84f5d23c10081042db722eb215c8af62441e3e2f6684\": container with ID starting with 2da91375ef30cbd47ccd84f5d23c10081042db722eb215c8af62441e3e2f6684 not found: ID does not exist" Nov 02 22:06:39.534196 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.534168 5871 scope.go:122] "RemoveContainer" containerID="dc00cab91a4160629f3a6a7df16ffbfa5feaa5d34c1f42c27d0a9f62996b5207" Nov 02 22:06:39.534464 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:39.534445 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc00cab91a4160629f3a6a7df16ffbfa5feaa5d34c1f42c27d0a9f62996b5207\": container with ID starting with dc00cab91a4160629f3a6a7df16ffbfa5feaa5d34c1f42c27d0a9f62996b5207 not found: ID does not exist" containerID="dc00cab91a4160629f3a6a7df16ffbfa5feaa5d34c1f42c27d0a9f62996b5207" Nov 02 22:06:39.534524 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.534476 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc00cab91a4160629f3a6a7df16ffbfa5feaa5d34c1f42c27d0a9f62996b5207"} err="failed to get container status \"dc00cab91a4160629f3a6a7df16ffbfa5feaa5d34c1f42c27d0a9f62996b5207\": rpc error: code = NotFound desc = could not find container \"dc00cab91a4160629f3a6a7df16ffbfa5feaa5d34c1f42c27d0a9f62996b5207\": container with ID starting with dc00cab91a4160629f3a6a7df16ffbfa5feaa5d34c1f42c27d0a9f62996b5207 not found: ID does not exist" Nov 02 22:06:39.534524 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.534501 5871 scope.go:122] "RemoveContainer" containerID="b0c2b65626f6d3796317124b61409892a23efe75bb852073d089f2314e02a792" Nov 02 22:06:39.534754 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:39.534737 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c2b65626f6d3796317124b61409892a23efe75bb852073d089f2314e02a792\": container with ID starting with b0c2b65626f6d3796317124b61409892a23efe75bb852073d089f2314e02a792 not found: ID does not exist" containerID="b0c2b65626f6d3796317124b61409892a23efe75bb852073d089f2314e02a792" Nov 02 22:06:39.534797 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.534759 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c2b65626f6d3796317124b61409892a23efe75bb852073d089f2314e02a792"} err="failed to get container status \"b0c2b65626f6d3796317124b61409892a23efe75bb852073d089f2314e02a792\": rpc error: code = NotFound desc = could not find container \"b0c2b65626f6d3796317124b61409892a23efe75bb852073d089f2314e02a792\": container with ID starting with b0c2b65626f6d3796317124b61409892a23efe75bb852073d089f2314e02a792 not found: ID does not exist" Nov 02 22:06:39.534797 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.534774 5871 scope.go:122] "RemoveContainer" containerID="4c6bfcbe1ed2682d3f93aa1b9f6cebed5f6034ac176331a328f14b458fe56582" Nov 02 22:06:39.535081 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:39.535055 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6bfcbe1ed2682d3f93aa1b9f6cebed5f6034ac176331a328f14b458fe56582\": container with ID starting with 4c6bfcbe1ed2682d3f93aa1b9f6cebed5f6034ac176331a328f14b458fe56582 not found: ID does not exist" containerID="4c6bfcbe1ed2682d3f93aa1b9f6cebed5f6034ac176331a328f14b458fe56582" Nov 02 22:06:39.535131 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.535093 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6bfcbe1ed2682d3f93aa1b9f6cebed5f6034ac176331a328f14b458fe56582"} err="failed to get container status \"4c6bfcbe1ed2682d3f93aa1b9f6cebed5f6034ac176331a328f14b458fe56582\": rpc error: code = NotFound desc = could not find container \"4c6bfcbe1ed2682d3f93aa1b9f6cebed5f6034ac176331a328f14b458fe56582\": container with ID starting with 4c6bfcbe1ed2682d3f93aa1b9f6cebed5f6034ac176331a328f14b458fe56582 not found: ID does not exist" Nov 02 22:06:39.535131 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.535115 5871 scope.go:122] "RemoveContainer" containerID="5891915245a856c716b3457d18b77ebc7a28274b31169b43ce4a646d3728ef74" Nov 02 22:06:39.535371 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:39.535353 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5891915245a856c716b3457d18b77ebc7a28274b31169b43ce4a646d3728ef74\": container with ID starting with 5891915245a856c716b3457d18b77ebc7a28274b31169b43ce4a646d3728ef74 not found: ID does not exist" containerID="5891915245a856c716b3457d18b77ebc7a28274b31169b43ce4a646d3728ef74" Nov 02 22:06:39.535413 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.535378 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5891915245a856c716b3457d18b77ebc7a28274b31169b43ce4a646d3728ef74"} err="failed to get container status \"5891915245a856c716b3457d18b77ebc7a28274b31169b43ce4a646d3728ef74\": rpc error: code = NotFound desc = could not find container \"5891915245a856c716b3457d18b77ebc7a28274b31169b43ce4a646d3728ef74\": container with ID starting with 5891915245a856c716b3457d18b77ebc7a28274b31169b43ce4a646d3728ef74 not found: ID does not exist" Nov 02 22:06:39.535413 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.535392 5871 scope.go:122] "RemoveContainer" containerID="b26d63ec309e5fe09be31f4a05a4bea233fd45eca992aa8e51fd2d6e209ed189" Nov 02 22:06:39.535687 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:39.535663 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26d63ec309e5fe09be31f4a05a4bea233fd45eca992aa8e51fd2d6e209ed189\": container with ID starting with b26d63ec309e5fe09be31f4a05a4bea233fd45eca992aa8e51fd2d6e209ed189 not found: ID does not exist" containerID="b26d63ec309e5fe09be31f4a05a4bea233fd45eca992aa8e51fd2d6e209ed189" Nov 02 22:06:39.535757 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.535695 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26d63ec309e5fe09be31f4a05a4bea233fd45eca992aa8e51fd2d6e209ed189"} err="failed to get container status \"b26d63ec309e5fe09be31f4a05a4bea233fd45eca992aa8e51fd2d6e209ed189\": rpc error: code = NotFound desc = could not find container \"b26d63ec309e5fe09be31f4a05a4bea233fd45eca992aa8e51fd2d6e209ed189\": container with ID starting with b26d63ec309e5fe09be31f4a05a4bea233fd45eca992aa8e51fd2d6e209ed189 not found: ID does not exist" Nov 02 22:06:39.535757 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.535718 5871 scope.go:122] "RemoveContainer" containerID="5064217c1727c131b0e57008af6162977bb22e38ebaf8db195cb216cf2f4ee4d" Nov 02 22:06:39.535980 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:39.535958 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5064217c1727c131b0e57008af6162977bb22e38ebaf8db195cb216cf2f4ee4d\": container with ID starting with 5064217c1727c131b0e57008af6162977bb22e38ebaf8db195cb216cf2f4ee4d not found: ID does not exist" containerID="5064217c1727c131b0e57008af6162977bb22e38ebaf8db195cb216cf2f4ee4d" Nov 02 22:06:39.536080 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.535985 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5064217c1727c131b0e57008af6162977bb22e38ebaf8db195cb216cf2f4ee4d"} err="failed to get container status \"5064217c1727c131b0e57008af6162977bb22e38ebaf8db195cb216cf2f4ee4d\": rpc error: code = NotFound desc = could not find container \"5064217c1727c131b0e57008af6162977bb22e38ebaf8db195cb216cf2f4ee4d\": container with ID starting with 5064217c1727c131b0e57008af6162977bb22e38ebaf8db195cb216cf2f4ee4d not found: ID does not exist" Nov 02 22:06:39.630121 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.630079 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:06:39.634124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:39.634094 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 22:06:40.563248 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:40.563217 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:40.746230 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:40.746198 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:41.311567 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:41.311536 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" path="/var/lib/kubelet/pods/a0d7bd65-c3af-49fa-9c77-8c7c348e73f4/volumes" Nov 02 22:06:41.495860 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:41.495816 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:41.496061 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:41.495942 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:41.501981 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:41.501949 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:42.489505 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:42.489474 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cc8bc6b4f-t55f8" Nov 02 22:06:42.647857 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:42.647819 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75d4dd99d4-vfls2"] Nov 02 22:06:43.401089 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.401016 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-59b677bf6c-t8dfh" podUID="e95934d1-f5df-4c06-ab39-5e7c8d58858b" containerName="console" containerID="cri-o://05ec70b6305730f6fbd0e9af1b2a15697a47251e2109fb1a219e7873db2c4c49" gracePeriod=15 Nov 02 22:06:43.653854 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.653790 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:43.737666 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.737631 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:06:43.737972 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.737948 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="prometheus" containerID="cri-o://b8be972e5e542e15ad9b2774d6cca6ce1d8089e98b82aa43824ee1e312852dc9" gracePeriod=600 Nov 02 22:06:43.738091 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.738058 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="thanos-sidecar" containerID="cri-o://8d41805cdf51542166697800adbd4e227ddf9b40058e0c71ac1c95230a3786c2" gracePeriod=600 Nov 02 22:06:43.738304 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.738093 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ce02962aeb3d3f93c6b43f04e4492a69a82ee40c0c6db38fd8a519ca0fced895" gracePeriod=600 Nov 02 22:06:43.738304 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.738052 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="kube-rbac-proxy" containerID="cri-o://b4b371d4ee20c71dc80ca2263fef811167c2e4a116ede6c1c6aeab345496b0fe" gracePeriod=600 Nov 02 22:06:43.738461 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.738090 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="config-reloader" containerID="cri-o://71c41332890264d27200d0629c0cb1a985229c1bcddd404c5796c5a816cdd45b" gracePeriod=600 Nov 02 22:06:43.738461 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.738069 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="kube-rbac-proxy-web" containerID="cri-o://58308a880679d6e6a966d64f4113567f63cfacd9c4bf3e000299bb1f5cdd0399" gracePeriod=600 Nov 02 22:06:43.773241 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:43.773203 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:06:43.791471 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.791443 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-trusted-ca-bundle\") pod \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " Nov 02 22:06:43.791604 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.791495 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e95934d1-f5df-4c06-ab39-5e7c8d58858b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e95934d1-f5df-4c06-ab39-5e7c8d58858b-tmp\") pod \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " Nov 02 22:06:43.791604 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.791540 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-serving-cert\") pod \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " Nov 02 22:06:43.791702 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.791630 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-config\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-config\") pod \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " Nov 02 22:06:43.791702 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.791661 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95934d1-f5df-4c06-ab39-5e7c8d58858b-kube-api-access-gbnh2\" (UniqueName: \"kubernetes.io/projected/e95934d1-f5df-4c06-ab39-5e7c8d58858b-kube-api-access-gbnh2\") pod \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " Nov 02 22:06:43.791810 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.791702 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-oauth-config\") pod \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " Nov 02 22:06:43.791810 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.791745 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-service-ca\") pod \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " Nov 02 22:06:43.791810 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.791777 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95934d1-f5df-4c06-ab39-5e7c8d58858b-tmp" pod "e95934d1-f5df-4c06-ab39-5e7c8d58858b" (UID: "e95934d1-f5df-4c06-ab39-5e7c8d58858b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:06:43.791810 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.791792 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-oauth-serving-cert\") pod \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\" (UID: \"e95934d1-f5df-4c06-ab39-5e7c8d58858b\") " Nov 02 22:06:43.792037 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.792017 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-trusted-ca-bundle" pod "e95934d1-f5df-4c06-ab39-5e7c8d58858b" (UID: "e95934d1-f5df-4c06-ab39-5e7c8d58858b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:43.792169 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.792142 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-oauth-serving-cert" pod "e95934d1-f5df-4c06-ab39-5e7c8d58858b" (UID: "e95934d1-f5df-4c06-ab39-5e7c8d58858b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:43.792300 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.792187 5871 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-trusted-ca-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:43.792300 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.792242 5871 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e95934d1-f5df-4c06-ab39-5e7c8d58858b-tmp\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:43.792300 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.792153 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-config" pod "e95934d1-f5df-4c06-ab39-5e7c8d58858b" (UID: "e95934d1-f5df-4c06-ab39-5e7c8d58858b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:43.792455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.792297 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-service-ca" pod "e95934d1-f5df-4c06-ab39-5e7c8d58858b" (UID: "e95934d1-f5df-4c06-ab39-5e7c8d58858b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:43.794137 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.794116 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95934d1-f5df-4c06-ab39-5e7c8d58858b-kube-api-access-gbnh2" pod "e95934d1-f5df-4c06-ab39-5e7c8d58858b" (UID: "e95934d1-f5df-4c06-ab39-5e7c8d58858b"). InnerVolumeSpecName "kube-api-access-gbnh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:06:43.794345 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.794313 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-serving-cert" pod "e95934d1-f5df-4c06-ab39-5e7c8d58858b" (UID: "e95934d1-f5df-4c06-ab39-5e7c8d58858b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:43.794437 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.794388 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-oauth-config" pod "e95934d1-f5df-4c06-ab39-5e7c8d58858b" (UID: "e95934d1-f5df-4c06-ab39-5e7c8d58858b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:43.893129 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.893096 5871 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-config\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:43.893298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.893151 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbnh2\" (UniqueName: \"kubernetes.io/projected/e95934d1-f5df-4c06-ab39-5e7c8d58858b-kube-api-access-gbnh2\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:43.893298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.893166 5871 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-oauth-config\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:43.893298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.893178 5871 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:43.893298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.893188 5871 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e95934d1-f5df-4c06-ab39-5e7c8d58858b-oauth-serving-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:43.893298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:43.893197 5871 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e95934d1-f5df-4c06-ab39-5e7c8d58858b-console-serving-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:44.495752 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.495720 5871 generic.go:356] "Generic (PLEG): container finished" podID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerID="ce02962aeb3d3f93c6b43f04e4492a69a82ee40c0c6db38fd8a519ca0fced895" exitCode=0 Nov 02 22:06:44.495752 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.495744 5871 generic.go:356] "Generic (PLEG): container finished" podID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerID="58308a880679d6e6a966d64f4113567f63cfacd9c4bf3e000299bb1f5cdd0399" exitCode=0 Nov 02 22:06:44.495752 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.495751 5871 generic.go:356] "Generic (PLEG): container finished" podID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerID="8d41805cdf51542166697800adbd4e227ddf9b40058e0c71ac1c95230a3786c2" exitCode=0 Nov 02 22:06:44.495752 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.495757 5871 generic.go:356] "Generic (PLEG): container finished" podID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerID="71c41332890264d27200d0629c0cb1a985229c1bcddd404c5796c5a816cdd45b" exitCode=0 Nov 02 22:06:44.496063 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.495791 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"272c657c-1cb5-4f20-a3b7-035c535c2498","Type":"ContainerDied","Data":"ce02962aeb3d3f93c6b43f04e4492a69a82ee40c0c6db38fd8a519ca0fced895"} Nov 02 22:06:44.496063 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.495862 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"272c657c-1cb5-4f20-a3b7-035c535c2498","Type":"ContainerDied","Data":"58308a880679d6e6a966d64f4113567f63cfacd9c4bf3e000299bb1f5cdd0399"} Nov 02 22:06:44.496063 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.495916 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"272c657c-1cb5-4f20-a3b7-035c535c2498","Type":"ContainerDied","Data":"8d41805cdf51542166697800adbd4e227ddf9b40058e0c71ac1c95230a3786c2"} Nov 02 22:06:44.496063 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.495946 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"272c657c-1cb5-4f20-a3b7-035c535c2498","Type":"ContainerDied","Data":"71c41332890264d27200d0629c0cb1a985229c1bcddd404c5796c5a816cdd45b"} Nov 02 22:06:44.496991 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.496969 5871 generic.go:356] "Generic (PLEG): container finished" podID="e95934d1-f5df-4c06-ab39-5e7c8d58858b" containerID="05ec70b6305730f6fbd0e9af1b2a15697a47251e2109fb1a219e7873db2c4c49" exitCode=0 Nov 02 22:06:44.497051 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.497014 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b677bf6c-t8dfh" Nov 02 22:06:44.497051 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.497027 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b677bf6c-t8dfh" event={"ID":"e95934d1-f5df-4c06-ab39-5e7c8d58858b","Type":"ContainerDied","Data":"05ec70b6305730f6fbd0e9af1b2a15697a47251e2109fb1a219e7873db2c4c49"} Nov 02 22:06:44.497120 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.497066 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b677bf6c-t8dfh" event={"ID":"e95934d1-f5df-4c06-ab39-5e7c8d58858b","Type":"ContainerDied","Data":"25e3292d0c67e6de4f616aafcfb0b0051f79b82ca336b9839f160f9a4298a8e6"} Nov 02 22:06:44.497120 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.497088 5871 scope.go:122] "RemoveContainer" containerID="05ec70b6305730f6fbd0e9af1b2a15697a47251e2109fb1a219e7873db2c4c49" Nov 02 22:06:44.506031 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.506011 5871 scope.go:122] "RemoveContainer" containerID="05ec70b6305730f6fbd0e9af1b2a15697a47251e2109fb1a219e7873db2c4c49" Nov 02 22:06:44.506409 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:44.506390 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ec70b6305730f6fbd0e9af1b2a15697a47251e2109fb1a219e7873db2c4c49\": container with ID starting with 05ec70b6305730f6fbd0e9af1b2a15697a47251e2109fb1a219e7873db2c4c49 not found: ID does not exist" containerID="05ec70b6305730f6fbd0e9af1b2a15697a47251e2109fb1a219e7873db2c4c49" Nov 02 22:06:44.506460 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.506421 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ec70b6305730f6fbd0e9af1b2a15697a47251e2109fb1a219e7873db2c4c49"} err="failed to get container status \"05ec70b6305730f6fbd0e9af1b2a15697a47251e2109fb1a219e7873db2c4c49\": rpc error: code = NotFound desc = could not find container \"05ec70b6305730f6fbd0e9af1b2a15697a47251e2109fb1a219e7873db2c4c49\": container with ID starting with 05ec70b6305730f6fbd0e9af1b2a15697a47251e2109fb1a219e7873db2c4c49 not found: ID does not exist" Nov 02 22:06:44.649587 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.649553 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59b677bf6c-t8dfh"] Nov 02 22:06:44.653599 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:44.653568 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59b677bf6c-t8dfh"] Nov 02 22:06:45.203447 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.203396 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6967dc6984-c2sqx" podUID="8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" containerName="console" containerID="cri-o://a2d1e014f1e3fc12e80006fda5de0de12b000056d6fd15a8742f1ba2d7f6c7b9" gracePeriod=15 Nov 02 22:06:45.310728 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.310696 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e95934d1-f5df-4c06-ab39-5e7c8d58858b" path="/var/lib/kubelet/pods/e95934d1-f5df-4c06-ab39-5e7c8d58858b/volumes" Nov 02 22:06:45.489536 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.489505 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:45.501745 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.501710 5871 generic.go:356] "Generic (PLEG): container finished" podID="8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" containerID="a2d1e014f1e3fc12e80006fda5de0de12b000056d6fd15a8742f1ba2d7f6c7b9" exitCode=0 Nov 02 22:06:45.501940 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.501761 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6967dc6984-c2sqx" Nov 02 22:06:45.501940 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.501791 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6967dc6984-c2sqx" event={"ID":"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1","Type":"ContainerDied","Data":"a2d1e014f1e3fc12e80006fda5de0de12b000056d6fd15a8742f1ba2d7f6c7b9"} Nov 02 22:06:45.501940 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.501840 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6967dc6984-c2sqx" event={"ID":"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1","Type":"ContainerDied","Data":"46409b905adf9a74d7a4145f7092b909ace522c82b374060c92da1474e1ce7f8"} Nov 02 22:06:45.501940 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.501864 5871 scope.go:122] "RemoveContainer" containerID="a2d1e014f1e3fc12e80006fda5de0de12b000056d6fd15a8742f1ba2d7f6c7b9" Nov 02 22:06:45.505708 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.505678 5871 generic.go:356] "Generic (PLEG): container finished" podID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerID="b4b371d4ee20c71dc80ca2263fef811167c2e4a116ede6c1c6aeab345496b0fe" exitCode=0 Nov 02 22:06:45.505840 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.505748 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"272c657c-1cb5-4f20-a3b7-035c535c2498","Type":"ContainerDied","Data":"b4b371d4ee20c71dc80ca2263fef811167c2e4a116ede6c1c6aeab345496b0fe"} Nov 02 22:06:45.510318 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.510301 5871 scope.go:122] "RemoveContainer" containerID="a2d1e014f1e3fc12e80006fda5de0de12b000056d6fd15a8742f1ba2d7f6c7b9" Nov 02 22:06:45.510609 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:45.510591 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d1e014f1e3fc12e80006fda5de0de12b000056d6fd15a8742f1ba2d7f6c7b9\": container with ID starting with a2d1e014f1e3fc12e80006fda5de0de12b000056d6fd15a8742f1ba2d7f6c7b9 not found: ID does not exist" containerID="a2d1e014f1e3fc12e80006fda5de0de12b000056d6fd15a8742f1ba2d7f6c7b9" Nov 02 22:06:45.510655 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.510620 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d1e014f1e3fc12e80006fda5de0de12b000056d6fd15a8742f1ba2d7f6c7b9"} err="failed to get container status \"a2d1e014f1e3fc12e80006fda5de0de12b000056d6fd15a8742f1ba2d7f6c7b9\": rpc error: code = NotFound desc = could not find container \"a2d1e014f1e3fc12e80006fda5de0de12b000056d6fd15a8742f1ba2d7f6c7b9\": container with ID starting with a2d1e014f1e3fc12e80006fda5de0de12b000056d6fd15a8742f1ba2d7f6c7b9 not found: ID does not exist" Nov 02 22:06:45.607167 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.607137 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-kube-api-access-8nbpz\" (UniqueName: \"kubernetes.io/projected/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-kube-api-access-8nbpz\") pod \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " Nov 02 22:06:45.607312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.607188 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-serving-cert\") pod \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " Nov 02 22:06:45.607312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.607208 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-oauth-serving-cert\") pod \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " Nov 02 22:06:45.607312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.607238 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-tmp\") pod \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " Nov 02 22:06:45.607312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.607266 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-config\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-config\") pod \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " Nov 02 22:06:45.607312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.607300 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-oauth-config\") pod \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " Nov 02 22:06:45.607588 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.607334 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-service-ca\") pod \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\" (UID: \"8a3102b4-bfe2-49a1-aa23-1c7b451f0da1\") " Nov 02 22:06:45.607646 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.607601 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-tmp" pod "8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" (UID: "8a3102b4-bfe2-49a1-aa23-1c7b451f0da1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:06:45.607759 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.607728 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-oauth-serving-cert" pod "8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" (UID: "8a3102b4-bfe2-49a1-aa23-1c7b451f0da1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:45.607902 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.607779 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-service-ca" pod "8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" (UID: "8a3102b4-bfe2-49a1-aa23-1c7b451f0da1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:45.608030 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.608004 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-config" pod "8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" (UID: "8a3102b4-bfe2-49a1-aa23-1c7b451f0da1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:45.609693 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.609660 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-kube-api-access-8nbpz" pod "8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" (UID: "8a3102b4-bfe2-49a1-aa23-1c7b451f0da1"). InnerVolumeSpecName "kube-api-access-8nbpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:06:45.609820 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.609800 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-oauth-config" pod "8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" (UID: "8a3102b4-bfe2-49a1-aa23-1c7b451f0da1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:45.609934 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.609911 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-serving-cert" pod "8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" (UID: "8a3102b4-bfe2-49a1-aa23-1c7b451f0da1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:45.708410 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.708319 5871 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-serving-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:45.708410 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.708354 5871 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-oauth-serving-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:45.708410 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.708368 5871 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-tmp\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:45.708410 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.708378 5871 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-config\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:45.708410 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.708386 5871 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-console-oauth-config\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:45.708410 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.708401 5871 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:45.708410 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.708412 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nbpz\" (UniqueName: \"kubernetes.io/projected/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1-kube-api-access-8nbpz\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:45.913676 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.913640 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6967dc6984-c2sqx"] Nov 02 22:06:45.917697 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:45.917658 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6967dc6984-c2sqx"] Nov 02 22:06:47.310780 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:47.310744 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" path="/var/lib/kubelet/pods/8a3102b4-bfe2-49a1-aa23-1c7b451f0da1/volumes" Nov 02 22:06:47.595773 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:47.595671 5871 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="b8be972e5e542e15ad9b2774d6cca6ce1d8089e98b82aa43824ee1e312852dc9" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Nov 02 22:06:47.595773 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:47.595742 5871 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="b8be972e5e542e15ad9b2774d6cca6ce1d8089e98b82aa43824ee1e312852dc9" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Nov 02 22:06:47.595773 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:06:47.595762 5871 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="b8be972e5e542e15ad9b2774d6cca6ce1d8089e98b82aa43824ee1e312852dc9" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Nov 02 22:06:47.596026 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:47.595776 5871 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled" Nov 02 22:06:49.524525 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:49.524492 5871 generic.go:356] "Generic (PLEG): container finished" podID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerID="b8be972e5e542e15ad9b2774d6cca6ce1d8089e98b82aa43824ee1e312852dc9" exitCode=0 Nov 02 22:06:49.524943 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:49.524547 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"272c657c-1cb5-4f20-a3b7-035c535c2498","Type":"ContainerDied","Data":"b8be972e5e542e15ad9b2774d6cca6ce1d8089e98b82aa43824ee1e312852dc9"} Nov 02 22:06:50.007309 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.007276 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-92hpx" Nov 02 22:06:50.026229 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.026196 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.150577 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.150489 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-config\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-config\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.150577 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.150535 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-1\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.150577 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.150563 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-kube-rbac-proxy\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.150911 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.150584 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-config-out\" (UniqueName: \"kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-config-out\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.150911 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.150610 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-web-config\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-web-config\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.150911 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.150636 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-metrics-client-certs\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.150911 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.150683 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-kube-api-access-dbw9d\" (UniqueName: \"kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-kube-api-access-dbw9d\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.150911 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.150722 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-tls-assets\" (UniqueName: \"kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-tls-assets\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.150911 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.150760 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.150911 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.150795 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.150911 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.150825 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-db\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.150911 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.150856 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-metrics-client-ca\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.151441 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.151227 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-metrics-client-ca" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:50.151441 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.151349 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-1" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:50.151887 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.151843 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-kubelet-serving-ca-bundle\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.151996 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.151925 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-2\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.151996 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.151964 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-thanos-prometheus-http-client-file\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.151996 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.151990 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-0\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.152164 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.152024 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-grpc-tls\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.152164 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.152051 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-tls\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.152164 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.152086 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-serving-certs-ca-bundle\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.152164 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.152111 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-trusted-ca-bundle\") pod \"272c657c-1cb5-4f20-a3b7-035c535c2498\" (UID: \"272c657c-1cb5-4f20-a3b7-035c535c2498\") " Nov 02 22:06:50.152369 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.152220 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-db" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:06:50.152450 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.152429 5871 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-db\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.152514 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.152459 5871 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-metrics-client-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.152514 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.152474 5871 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.153724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.153430 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-2" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:50.153724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.153469 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-trusted-ca-bundle" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:50.153724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.153555 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-serving-certs-ca-bundle" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:50.153986 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.153784 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-kubelet-serving-ca-bundle" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:50.155389 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.155356 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-0" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:06:50.156179 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.156122 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-config" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:50.157405 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.157364 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-kube-api-access-dbw9d" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "kube-api-access-dbw9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:06:50.157405 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.157385 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-thanos-sidecar-tls" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:50.157560 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.157430 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-tls-assets" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:06:50.157560 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.157439 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-config-out" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:06:50.157560 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.157524 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-metrics-client-certs" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:50.157788 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.157701 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-grpc-tls" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:50.157958 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.157923 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-tls" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:50.158033 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.157969 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-thanos-prometheus-http-client-file" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:50.158096 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.158037 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-kube-rbac-proxy-web" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:50.158230 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.158210 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-kube-rbac-proxy" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:50.167853 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.167827 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-web-config" pod "272c657c-1cb5-4f20-a3b7-035c535c2498" (UID: "272c657c-1cb5-4f20-a3b7-035c535c2498"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:06:50.253686 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253646 5871 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.253686 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253678 5871 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.253686 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253689 5871 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.253964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253699 5871 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.253964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253709 5871 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-grpc-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.253964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253719 5871 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.253964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253729 5871 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.253964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253737 5871 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/272c657c-1cb5-4f20-a3b7-035c535c2498-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.253964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253746 5871 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-config\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.253964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253755 5871 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-kube-rbac-proxy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.253964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253764 5871 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/272c657c-1cb5-4f20-a3b7-035c535c2498-config-out\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.253964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253774 5871 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-web-config\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.253964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253782 5871 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-metrics-client-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.253964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253791 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbw9d\" (UniqueName: \"kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-kube-api-access-dbw9d\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.253964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253800 5871 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/272c657c-1cb5-4f20-a3b7-035c535c2498-tls-assets\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.253964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253808 5871 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.253964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.253820 5871 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/272c657c-1cb5-4f20-a3b7-035c535c2498-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:06:50.532223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.532144 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"272c657c-1cb5-4f20-a3b7-035c535c2498","Type":"ContainerDied","Data":"be03c1c4bf106b4e190ff5861d9f12fb41fd9db1b7bbca2dc2ab18a2bee7bdca"} Nov 02 22:06:50.532223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.532189 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.532223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.532221 5871 scope.go:122] "RemoveContainer" containerID="ce02962aeb3d3f93c6b43f04e4492a69a82ee40c0c6db38fd8a519ca0fced895" Nov 02 22:06:50.540085 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.540069 5871 scope.go:122] "RemoveContainer" containerID="b4b371d4ee20c71dc80ca2263fef811167c2e4a116ede6c1c6aeab345496b0fe" Nov 02 22:06:50.547204 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.547187 5871 scope.go:122] "RemoveContainer" containerID="58308a880679d6e6a966d64f4113567f63cfacd9c4bf3e000299bb1f5cdd0399" Nov 02 22:06:50.554540 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.554519 5871 scope.go:122] "RemoveContainer" containerID="8d41805cdf51542166697800adbd4e227ddf9b40058e0c71ac1c95230a3786c2" Nov 02 22:06:50.561833 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.561808 5871 scope.go:122] "RemoveContainer" containerID="71c41332890264d27200d0629c0cb1a985229c1bcddd404c5796c5a816cdd45b" Nov 02 22:06:50.570640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.570613 5871 scope.go:122] "RemoveContainer" containerID="b8be972e5e542e15ad9b2774d6cca6ce1d8089e98b82aa43824ee1e312852dc9" Nov 02 22:06:50.579216 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.579196 5871 scope.go:122] "RemoveContainer" containerID="368273d4a42d7eae3ebb350aa7b1a269bcedd1b8a57eefb77aa54f03d4f5d6d1" Nov 02 22:06:50.696140 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.693946 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:06:50.738594 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.738556 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:06:50.738942 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.738920 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:06:50.739197 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739184 5871 cpu_manager.go:414] "containerMap: removing container" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="kube-rbac-proxy-web" Nov 02 22:06:50.739234 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739198 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="kube-rbac-proxy-web" Nov 02 22:06:50.739234 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739209 5871 cpu_manager.go:414] "containerMap: removing container" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="config-reloader" Nov 02 22:06:50.739234 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739213 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="config-reloader" Nov 02 22:06:50.739234 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739221 5871 cpu_manager.go:414] "containerMap: removing container" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="prom-label-proxy" Nov 02 22:06:50.739234 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739226 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="prom-label-proxy" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739236 5871 cpu_manager.go:414] "containerMap: removing container" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="init-config-reloader" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739241 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="init-config-reloader" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739248 5871 cpu_manager.go:414] "containerMap: removing container" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="kube-rbac-proxy" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739253 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="kube-rbac-proxy" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739265 5871 cpu_manager.go:414] "containerMap: removing container" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="kube-rbac-proxy" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739270 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="kube-rbac-proxy" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739277 5871 cpu_manager.go:414] "containerMap: removing container" podUID="e95934d1-f5df-4c06-ab39-5e7c8d58858b" containerName="console" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739281 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e95934d1-f5df-4c06-ab39-5e7c8d58858b" containerName="console" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739288 5871 cpu_manager.go:414] "containerMap: removing container" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="kube-rbac-proxy-thanos" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739293 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="kube-rbac-proxy-thanos" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739300 5871 cpu_manager.go:414] "containerMap: removing container" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="kube-rbac-proxy-web" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739305 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="kube-rbac-proxy-web" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739314 5871 cpu_manager.go:414] "containerMap: removing container" podUID="8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" containerName="console" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739319 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" containerName="console" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739328 5871 cpu_manager.go:414] "containerMap: removing container" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="init-config-reloader" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739332 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="init-config-reloader" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739339 5871 cpu_manager.go:414] "containerMap: removing container" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="thanos-sidecar" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739343 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="thanos-sidecar" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739350 5871 cpu_manager.go:414] "containerMap: removing container" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="config-reloader" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739355 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="config-reloader" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739360 5871 cpu_manager.go:414] "containerMap: removing container" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="alertmanager" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739364 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="alertmanager" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739371 5871 cpu_manager.go:414] "containerMap: removing container" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="kube-rbac-proxy-metric" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739376 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="kube-rbac-proxy-metric" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739383 5871 cpu_manager.go:414] "containerMap: removing container" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="prometheus" Nov 02 22:06:50.739379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739388 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="prometheus" Nov 02 22:06:50.740352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739435 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="kube-rbac-proxy" Nov 02 22:06:50.740352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739442 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="e95934d1-f5df-4c06-ab39-5e7c8d58858b" containerName="console" Nov 02 22:06:50.740352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739448 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="kube-rbac-proxy-thanos" Nov 02 22:06:50.740352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739455 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="kube-rbac-proxy-web" Nov 02 22:06:50.740352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739460 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3102b4-bfe2-49a1-aa23-1c7b451f0da1" containerName="console" Nov 02 22:06:50.740352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739466 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="init-config-reloader" Nov 02 22:06:50.740352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739470 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="thanos-sidecar" Nov 02 22:06:50.740352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739475 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="config-reloader" Nov 02 22:06:50.740352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739480 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="alertmanager" Nov 02 22:06:50.740352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739485 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="kube-rbac-proxy-metric" Nov 02 22:06:50.740352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739489 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="prometheus" Nov 02 22:06:50.740352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739494 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="kube-rbac-proxy-web" Nov 02 22:06:50.740352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739498 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="config-reloader" Nov 02 22:06:50.740352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739507 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="prom-label-proxy" Nov 02 22:06:50.740352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739512 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" containerName="init-config-reloader" Nov 02 22:06:50.740352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.739517 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d7bd65-c3af-49fa-9c77-8c7c348e73f4" containerName="kube-rbac-proxy" Nov 02 22:06:50.771823 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.771782 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.820947 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.820817 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 22:06:50.821275 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.821253 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 22:06:50.822158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.821598 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 22:06:50.822158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.821598 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 22:06:50.822158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.821627 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 22:06:50.822158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.821641 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 22:06:50.822158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.821609 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 22:06:50.822158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.821686 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 22:06:50.822158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.821817 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 22:06:50.822158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.821849 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 22:06:50.822158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.821915 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 22:06:50.822158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.822051 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vgqck\"" Nov 02 22:06:50.822158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.822096 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dmih5pep9rcoc\"" Nov 02 22:06:50.822158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.822100 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 22:06:50.856632 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.856595 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-config\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.856632 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.856633 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.856707 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.856744 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.856843 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.856933 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.856980 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.857018 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.857042 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857432 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.857072 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857432 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.857097 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857432 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.857138 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857432 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.857164 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857432 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.857194 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857432 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.857242 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-web-config\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857432 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.857267 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-config-out\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857432 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.857292 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857432 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.857315 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcqjp\" (UniqueName: \"kubernetes.io/projected/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-kube-api-access-qcqjp\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857432 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.857363 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.857432 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.857385 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.868799 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.868771 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 22:06:50.868799 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.868785 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 22:06:50.869460 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.869441 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:06:50.957846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.957807 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.957846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.957848 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958127 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.957896 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958127 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.957916 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958127 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.957940 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958127 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.957970 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-web-config\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958127 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.958001 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-config-out\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958127 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.958029 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958127 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.958053 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcqjp\" (UniqueName: \"kubernetes.io/projected/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-kube-api-access-qcqjp\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958127 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.958108 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958127 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.958132 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958542 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.958156 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-config\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958542 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.958172 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958542 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.958195 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958542 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.958221 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958542 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.958246 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958542 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.958273 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958542 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.958313 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958542 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.958349 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.958542 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.958373 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.959061 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.958734 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.959316 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.959286 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.959811 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.959787 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.960903 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.960849 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.961289 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.961269 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.961458 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.961433 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.961541 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.961477 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.961694 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.961653 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.962956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.962278 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-config-out\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.962956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.962506 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.962956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.962660 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-web-config\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.962956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.962776 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.962956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.962910 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.963272 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.963057 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.964107 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.964080 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-config\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.964201 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.964133 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.964201 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.964150 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.964285 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.964224 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:50.964859 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:50.964839 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:51.018037 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:51.018005 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcqjp\" (UniqueName: \"kubernetes.io/projected/ac460a0c-d54c-4ce4-b184-84c6c00a5db7-kube-api-access-qcqjp\") pod \"prometheus-k8s-0\" (UID: \"ac460a0c-d54c-4ce4-b184-84c6c00a5db7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:51.177860 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:51.177812 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:06:51.310884 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:51.310835 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="272c657c-1cb5-4f20-a3b7-035c535c2498" path="/var/lib/kubelet/pods/272c657c-1cb5-4f20-a3b7-035c535c2498/volumes" Nov 02 22:06:51.662329 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:51.662276 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 22:06:51.671260 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:06:51.671230 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac460a0c_d54c_4ce4_b184_84c6c00a5db7.slice/crio-33984a491f152812d66e40c9a012b95925f93c1eb9f4e912db403c81a41879fc WatchSource:0}: Error finding container 33984a491f152812d66e40c9a012b95925f93c1eb9f4e912db403c81a41879fc: Status 404 returned error can't find the container with id 33984a491f152812d66e40c9a012b95925f93c1eb9f4e912db403c81a41879fc Nov 02 22:06:52.541845 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:52.541811 5871 generic.go:356] "Generic (PLEG): container finished" podID="ac460a0c-d54c-4ce4-b184-84c6c00a5db7" containerID="2fa694a53c05f05130728993c4c11598de753c2a6f9cd697510b11c29c3c0143" exitCode=0 Nov 02 22:06:52.542032 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:52.541913 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac460a0c-d54c-4ce4-b184-84c6c00a5db7","Type":"ContainerDied","Data":"2fa694a53c05f05130728993c4c11598de753c2a6f9cd697510b11c29c3c0143"} Nov 02 22:06:52.542032 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:52.541986 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac460a0c-d54c-4ce4-b184-84c6c00a5db7","Type":"ContainerStarted","Data":"33984a491f152812d66e40c9a012b95925f93c1eb9f4e912db403c81a41879fc"} Nov 02 22:06:53.549400 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:53.549359 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac460a0c-d54c-4ce4-b184-84c6c00a5db7","Type":"ContainerStarted","Data":"940f735e8d63017230cf14473cce82f5de4da30ea4e4cfda7d9d6715de91764a"} Nov 02 22:06:53.549833 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:53.549439 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac460a0c-d54c-4ce4-b184-84c6c00a5db7","Type":"ContainerStarted","Data":"e59bcd8a1387abba3ae81ade1a69657a2aca046ed47b94d0a1280087b77a7653"} Nov 02 22:06:53.549833 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:53.549481 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac460a0c-d54c-4ce4-b184-84c6c00a5db7","Type":"ContainerStarted","Data":"afd72fd7b749aaa8b250eecfabc62b18a9ad96d015167eadbeeaba2d34745e64"} Nov 02 22:06:53.549833 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:53.549520 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac460a0c-d54c-4ce4-b184-84c6c00a5db7","Type":"ContainerStarted","Data":"51b51da49838b5214a3d5edbce2ab89dfbffd8f6febf9f1978f80b3d5de88582"} Nov 02 22:06:53.549833 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:53.549566 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac460a0c-d54c-4ce4-b184-84c6c00a5db7","Type":"ContainerStarted","Data":"07ad7654e3ce7af453d6dcf3bcc2dcb799ab706a30aaecce070f5a53d5ce3ce4"} Nov 02 22:06:53.549833 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:53.549604 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ac460a0c-d54c-4ce4-b184-84c6c00a5db7","Type":"ContainerStarted","Data":"bf824827b9b93465c82befbf7d5003f9c677c7449f8bf815342f2318eee95aee"} Nov 02 22:06:53.674925 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:53.674838 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.657714968 podStartE2EDuration="3.674817525s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.102557ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:53.673593568 +0000 UTC m=+36.485923747" watchObservedRunningTime="2026-11-02 22:06:53.674817525 +0000 UTC m=+36.487147706" Nov 02 22:06:54.656405 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:54.656367 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:55.579596 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:55.579559 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:06:56.178228 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:06:56.178192 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:07:07.668226 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:07.668185 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-75d4dd99d4-vfls2" podUID="7d6c8f46-7560-45fb-8d07-f32136e14bce" containerName="console" containerID="cri-o://68310839cfeb8481a4b50be3fda419859d1dcc406f30ec4659c5ad362807667b" gracePeriod=15 Nov 02 22:07:07.919989 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:07.919923 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:07:08.005431 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.005386 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-oauth-config\") pod \"7d6c8f46-7560-45fb-8d07-f32136e14bce\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " Nov 02 22:07:08.005605 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.005472 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6c8f46-7560-45fb-8d07-f32136e14bce-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d6c8f46-7560-45fb-8d07-f32136e14bce-tmp\") pod \"7d6c8f46-7560-45fb-8d07-f32136e14bce\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " Nov 02 22:07:08.005605 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.005538 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-service-ca\") pod \"7d6c8f46-7560-45fb-8d07-f32136e14bce\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " Nov 02 22:07:08.005605 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.005582 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-serving-cert\") pod \"7d6c8f46-7560-45fb-8d07-f32136e14bce\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " Nov 02 22:07:08.005744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.005609 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-oauth-serving-cert\") pod \"7d6c8f46-7560-45fb-8d07-f32136e14bce\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " Nov 02 22:07:08.005744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.005647 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-config\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-config\") pod \"7d6c8f46-7560-45fb-8d07-f32136e14bce\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " Nov 02 22:07:08.005744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.005692 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6c8f46-7560-45fb-8d07-f32136e14bce-kube-api-access-5p2xh\" (UniqueName: \"kubernetes.io/projected/7d6c8f46-7560-45fb-8d07-f32136e14bce-kube-api-access-5p2xh\") pod \"7d6c8f46-7560-45fb-8d07-f32136e14bce\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " Nov 02 22:07:08.005744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.005721 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-trusted-ca-bundle\") pod \"7d6c8f46-7560-45fb-8d07-f32136e14bce\" (UID: \"7d6c8f46-7560-45fb-8d07-f32136e14bce\") " Nov 02 22:07:08.005984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.005722 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6c8f46-7560-45fb-8d07-f32136e14bce-tmp" pod "7d6c8f46-7560-45fb-8d07-f32136e14bce" (UID: "7d6c8f46-7560-45fb-8d07-f32136e14bce"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:07:08.006051 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.006001 5871 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d6c8f46-7560-45fb-8d07-f32136e14bce-tmp\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:07:08.006107 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.006054 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-service-ca" pod "7d6c8f46-7560-45fb-8d07-f32136e14bce" (UID: "7d6c8f46-7560-45fb-8d07-f32136e14bce"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:08.006166 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.006106 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-oauth-serving-cert" pod "7d6c8f46-7560-45fb-8d07-f32136e14bce" (UID: "7d6c8f46-7560-45fb-8d07-f32136e14bce"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:08.006166 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.006123 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-config" pod "7d6c8f46-7560-45fb-8d07-f32136e14bce" (UID: "7d6c8f46-7560-45fb-8d07-f32136e14bce"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:08.006406 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.006370 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-trusted-ca-bundle" pod "7d6c8f46-7560-45fb-8d07-f32136e14bce" (UID: "7d6c8f46-7560-45fb-8d07-f32136e14bce"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:07:08.007913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.007886 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-oauth-config" pod "7d6c8f46-7560-45fb-8d07-f32136e14bce" (UID: "7d6c8f46-7560-45fb-8d07-f32136e14bce"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:07:08.007913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.007899 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6c8f46-7560-45fb-8d07-f32136e14bce-kube-api-access-5p2xh" pod "7d6c8f46-7560-45fb-8d07-f32136e14bce" (UID: "7d6c8f46-7560-45fb-8d07-f32136e14bce"). InnerVolumeSpecName "kube-api-access-5p2xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:07:08.008102 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.007976 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-serving-cert" pod "7d6c8f46-7560-45fb-8d07-f32136e14bce" (UID: "7d6c8f46-7560-45fb-8d07-f32136e14bce"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:07:08.106803 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.106761 5871 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-oauth-config\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:07:08.106803 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.106801 5871 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:07:08.106803 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.106811 5871 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-serving-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:07:08.107068 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.106819 5871 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-oauth-serving-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:07:08.107068 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.106828 5871 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-console-config\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:07:08.107068 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.106837 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p2xh\" (UniqueName: \"kubernetes.io/projected/7d6c8f46-7560-45fb-8d07-f32136e14bce-kube-api-access-5p2xh\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:07:08.107068 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.106848 5871 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d6c8f46-7560-45fb-8d07-f32136e14bce-trusted-ca-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:07:08.604491 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.604454 5871 generic.go:356] "Generic (PLEG): container finished" podID="7d6c8f46-7560-45fb-8d07-f32136e14bce" containerID="68310839cfeb8481a4b50be3fda419859d1dcc406f30ec4659c5ad362807667b" exitCode=0 Nov 02 22:07:08.604677 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.604507 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d4dd99d4-vfls2" Nov 02 22:07:08.604677 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.604541 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d4dd99d4-vfls2" event={"ID":"7d6c8f46-7560-45fb-8d07-f32136e14bce","Type":"ContainerDied","Data":"68310839cfeb8481a4b50be3fda419859d1dcc406f30ec4659c5ad362807667b"} Nov 02 22:07:08.604677 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.604597 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d4dd99d4-vfls2" event={"ID":"7d6c8f46-7560-45fb-8d07-f32136e14bce","Type":"ContainerDied","Data":"85140f1873aedf163a05981c5444c605f5659b20ff33594380bb0316f5f7cbd7"} Nov 02 22:07:08.604677 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.604618 5871 scope.go:122] "RemoveContainer" containerID="68310839cfeb8481a4b50be3fda419859d1dcc406f30ec4659c5ad362807667b" Nov 02 22:07:08.615348 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.615327 5871 scope.go:122] "RemoveContainer" containerID="68310839cfeb8481a4b50be3fda419859d1dcc406f30ec4659c5ad362807667b" Nov 02 22:07:08.615717 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:07:08.615696 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68310839cfeb8481a4b50be3fda419859d1dcc406f30ec4659c5ad362807667b\": container with ID starting with 68310839cfeb8481a4b50be3fda419859d1dcc406f30ec4659c5ad362807667b not found: ID does not exist" containerID="68310839cfeb8481a4b50be3fda419859d1dcc406f30ec4659c5ad362807667b" Nov 02 22:07:08.615773 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.615729 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68310839cfeb8481a4b50be3fda419859d1dcc406f30ec4659c5ad362807667b"} err="failed to get container status \"68310839cfeb8481a4b50be3fda419859d1dcc406f30ec4659c5ad362807667b\": rpc error: code = NotFound desc = could not find container \"68310839cfeb8481a4b50be3fda419859d1dcc406f30ec4659c5ad362807667b\": container with ID starting with 68310839cfeb8481a4b50be3fda419859d1dcc406f30ec4659c5ad362807667b not found: ID does not exist" Nov 02 22:07:08.757505 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.757469 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75d4dd99d4-vfls2"] Nov 02 22:07:08.803543 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:08.803500 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75d4dd99d4-vfls2"] Nov 02 22:07:09.311100 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:09.311066 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d6c8f46-7560-45fb-8d07-f32136e14bce" path="/var/lib/kubelet/pods/7d6c8f46-7560-45fb-8d07-f32136e14bce/volumes" Nov 02 22:07:13.769260 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:07:13.769225 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:07:14.246474 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:14.246434 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:07:14.262292 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:14.262263 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:07:14.642533 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:14.642505 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:07:18.144283 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:07:18.144241 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:07:43.770242 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:07:43.770199 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:08:08.386193 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:08:08.386099 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:13.769742 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:08:13.769703 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:08:18.495162 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:08:18.495112 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:31.890082 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:08:31.890051 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:43.770004 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:08:43.769960 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:08:58.144668 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:08:58.144634 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:58.332444 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:08:58.332403 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:59.702179 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:08:59.702143 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:08:59.883999 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:08:59.883946 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:01.073236 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:01.073193 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:02.548011 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.547978 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs"] Nov 02 22:09:02.548484 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.548253 5871 cpu_manager.go:414] "containerMap: removing container" podUID="7d6c8f46-7560-45fb-8d07-f32136e14bce" containerName="console" Nov 02 22:09:02.548484 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.548264 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6c8f46-7560-45fb-8d07-f32136e14bce" containerName="console" Nov 02 22:09:02.548484 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.548313 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6c8f46-7560-45fb-8d07-f32136e14bce" containerName="console" Nov 02 22:09:02.550659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.550632 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" Nov 02 22:09:02.631972 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.631933 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n2sd\" (UniqueName: \"kubernetes.io/projected/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-kube-api-access-5n2sd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs\" (UID: \"0174771e-0c4e-4d4d-9f5f-a260a6719f1a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" Nov 02 22:09:02.632156 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.631994 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs\" (UID: \"0174771e-0c4e-4d4d-9f5f-a260a6719f1a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" Nov 02 22:09:02.632156 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.632067 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs\" (UID: \"0174771e-0c4e-4d4d-9f5f-a260a6719f1a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" Nov 02 22:09:02.645983 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.645944 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 22:09:02.646052 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.645996 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-gccj8\"" Nov 02 22:09:02.646052 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.646029 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 22:09:02.646642 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.646622 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs"] Nov 02 22:09:02.732723 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.732679 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n2sd\" (UniqueName: \"kubernetes.io/projected/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-kube-api-access-5n2sd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs\" (UID: \"0174771e-0c4e-4d4d-9f5f-a260a6719f1a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" Nov 02 22:09:02.732921 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.732738 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs\" (UID: \"0174771e-0c4e-4d4d-9f5f-a260a6719f1a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" Nov 02 22:09:02.732921 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.732785 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs\" (UID: \"0174771e-0c4e-4d4d-9f5f-a260a6719f1a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" Nov 02 22:09:02.733208 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.733185 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs\" (UID: \"0174771e-0c4e-4d4d-9f5f-a260a6719f1a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" Nov 02 22:09:02.733253 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.733196 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs\" (UID: \"0174771e-0c4e-4d4d-9f5f-a260a6719f1a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" Nov 02 22:09:02.790544 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.790507 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n2sd\" (UniqueName: \"kubernetes.io/projected/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-kube-api-access-5n2sd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs\" (UID: \"0174771e-0c4e-4d4d-9f5f-a260a6719f1a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" Nov 02 22:09:02.863530 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:02.863499 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" Nov 02 22:09:03.237933 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.237816 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs"] Nov 02 22:09:03.249812 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.249791 5871 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:09:03.579610 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.579524 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml"] Nov 02 22:09:03.582947 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.582926 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" Nov 02 22:09:03.646550 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.640968 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grqr9\" (UniqueName: \"kubernetes.io/projected/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-kube-api-access-grqr9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml\" (UID: \"6e3e8158-f5c0-4dc2-80b1-257e1ee30487\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" Nov 02 22:09:03.646550 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.641104 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml\" (UID: \"6e3e8158-f5c0-4dc2-80b1-257e1ee30487\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" Nov 02 22:09:03.646550 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.641163 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml\" (UID: \"6e3e8158-f5c0-4dc2-80b1-257e1ee30487\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" Nov 02 22:09:03.678138 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.678100 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-5zkgr\"" Nov 02 22:09:03.678837 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.678812 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml"] Nov 02 22:09:03.681082 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.681045 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g"] Nov 02 22:09:03.683985 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.683964 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" Nov 02 22:09:03.733402 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.733363 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-vh5f8\"" Nov 02 22:09:03.742299 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.742270 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml\" (UID: \"6e3e8158-f5c0-4dc2-80b1-257e1ee30487\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" Nov 02 22:09:03.742387 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.742323 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml\" (UID: \"6e3e8158-f5c0-4dc2-80b1-257e1ee30487\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" Nov 02 22:09:03.742433 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.742351 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90a9d920-ed37-4c77-89e6-9c5396dff5e2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g\" (UID: \"90a9d920-ed37-4c77-89e6-9c5396dff5e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" Nov 02 22:09:03.742433 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.742424 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grqr9\" (UniqueName: \"kubernetes.io/projected/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-kube-api-access-grqr9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml\" (UID: \"6e3e8158-f5c0-4dc2-80b1-257e1ee30487\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" Nov 02 22:09:03.742510 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.742444 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf7kb\" (UniqueName: \"kubernetes.io/projected/90a9d920-ed37-4c77-89e6-9c5396dff5e2-kube-api-access-sf7kb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g\" (UID: \"90a9d920-ed37-4c77-89e6-9c5396dff5e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" Nov 02 22:09:03.742510 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.742471 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90a9d920-ed37-4c77-89e6-9c5396dff5e2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g\" (UID: \"90a9d920-ed37-4c77-89e6-9c5396dff5e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" Nov 02 22:09:03.742752 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.742718 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml\" (UID: \"6e3e8158-f5c0-4dc2-80b1-257e1ee30487\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" Nov 02 22:09:03.742831 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.742784 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml\" (UID: \"6e3e8158-f5c0-4dc2-80b1-257e1ee30487\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" Nov 02 22:09:03.744048 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.744023 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g"] Nov 02 22:09:03.780783 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.780740 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8"] Nov 02 22:09:03.783546 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.783526 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" Nov 02 22:09:03.799971 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.799942 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grqr9\" (UniqueName: \"kubernetes.io/projected/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-kube-api-access-grqr9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml\" (UID: \"6e3e8158-f5c0-4dc2-80b1-257e1ee30487\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" Nov 02 22:09:03.833607 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.833518 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-qkz7t\"" Nov 02 22:09:03.839809 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.839779 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8"] Nov 02 22:09:03.843019 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.842991 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90a9d920-ed37-4c77-89e6-9c5396dff5e2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g\" (UID: \"90a9d920-ed37-4c77-89e6-9c5396dff5e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" Nov 02 22:09:03.843169 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.843030 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf7kb\" (UniqueName: \"kubernetes.io/projected/90a9d920-ed37-4c77-89e6-9c5396dff5e2-kube-api-access-sf7kb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g\" (UID: \"90a9d920-ed37-4c77-89e6-9c5396dff5e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" Nov 02 22:09:03.843169 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.843058 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bflws\" (UniqueName: \"kubernetes.io/projected/f01be6b4-9954-414a-a080-1a438c467ff3-kube-api-access-bflws\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8\" (UID: \"f01be6b4-9954-414a-a080-1a438c467ff3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" Nov 02 22:09:03.843169 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.843087 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f01be6b4-9954-414a-a080-1a438c467ff3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8\" (UID: \"f01be6b4-9954-414a-a080-1a438c467ff3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" Nov 02 22:09:03.843169 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.843118 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90a9d920-ed37-4c77-89e6-9c5396dff5e2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g\" (UID: \"90a9d920-ed37-4c77-89e6-9c5396dff5e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" Nov 02 22:09:03.843368 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.843300 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f01be6b4-9954-414a-a080-1a438c467ff3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8\" (UID: \"f01be6b4-9954-414a-a080-1a438c467ff3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" Nov 02 22:09:03.843694 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.843675 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90a9d920-ed37-4c77-89e6-9c5396dff5e2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g\" (UID: \"90a9d920-ed37-4c77-89e6-9c5396dff5e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" Nov 02 22:09:03.843793 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.843732 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90a9d920-ed37-4c77-89e6-9c5396dff5e2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g\" (UID: \"90a9d920-ed37-4c77-89e6-9c5396dff5e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" Nov 02 22:09:03.881586 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.881542 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs"] Nov 02 22:09:03.884524 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.884498 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" Nov 02 22:09:03.896399 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.896374 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" Nov 02 22:09:03.901294 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.901269 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf7kb\" (UniqueName: \"kubernetes.io/projected/90a9d920-ed37-4c77-89e6-9c5396dff5e2-kube-api-access-sf7kb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g\" (UID: \"90a9d920-ed37-4c77-89e6-9c5396dff5e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" Nov 02 22:09:03.933920 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.933855 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-wtkgg\"" Nov 02 22:09:03.941313 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.941218 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs"] Nov 02 22:09:03.945841 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.943772 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs\" (UID: \"6f310886-2c25-4b42-b71c-2bbb20b2a4eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" Nov 02 22:09:03.945841 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.943841 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bflws\" (UniqueName: \"kubernetes.io/projected/f01be6b4-9954-414a-a080-1a438c467ff3-kube-api-access-bflws\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8\" (UID: \"f01be6b4-9954-414a-a080-1a438c467ff3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" Nov 02 22:09:03.945841 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.943892 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f01be6b4-9954-414a-a080-1a438c467ff3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8\" (UID: \"f01be6b4-9954-414a-a080-1a438c467ff3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" Nov 02 22:09:03.945841 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.943934 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sppn6\" (UniqueName: \"kubernetes.io/projected/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-kube-api-access-sppn6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs\" (UID: \"6f310886-2c25-4b42-b71c-2bbb20b2a4eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" Nov 02 22:09:03.945841 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.943972 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs\" (UID: \"6f310886-2c25-4b42-b71c-2bbb20b2a4eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" Nov 02 22:09:03.945841 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.944028 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f01be6b4-9954-414a-a080-1a438c467ff3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8\" (UID: \"f01be6b4-9954-414a-a080-1a438c467ff3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" Nov 02 22:09:03.945841 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.944501 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f01be6b4-9954-414a-a080-1a438c467ff3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8\" (UID: \"f01be6b4-9954-414a-a080-1a438c467ff3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" Nov 02 22:09:03.947497 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.947473 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f01be6b4-9954-414a-a080-1a438c467ff3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8\" (UID: \"f01be6b4-9954-414a-a080-1a438c467ff3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" Nov 02 22:09:03.982237 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.982202 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" event={"ID":"0174771e-0c4e-4d4d-9f5f-a260a6719f1a","Type":"ContainerStarted","Data":"5ecf3c25a9bf3f760e5b37d22df73decc433f36f002f7bc46f80636c0fc88391"} Nov 02 22:09:03.997470 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:03.997432 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" Nov 02 22:09:04.002817 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.002779 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bflws\" (UniqueName: \"kubernetes.io/projected/f01be6b4-9954-414a-a080-1a438c467ff3-kube-api-access-bflws\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8\" (UID: \"f01be6b4-9954-414a-a080-1a438c467ff3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" Nov 02 22:09:04.044583 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.044533 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppn6\" (UniqueName: \"kubernetes.io/projected/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-kube-api-access-sppn6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs\" (UID: \"6f310886-2c25-4b42-b71c-2bbb20b2a4eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" Nov 02 22:09:04.044763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.044596 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs\" (UID: \"6f310886-2c25-4b42-b71c-2bbb20b2a4eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" Nov 02 22:09:04.044763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.044701 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs\" (UID: \"6f310886-2c25-4b42-b71c-2bbb20b2a4eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" Nov 02 22:09:04.045186 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.045160 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs\" (UID: \"6f310886-2c25-4b42-b71c-2bbb20b2a4eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" Nov 02 22:09:04.046827 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.045410 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs\" (UID: \"6f310886-2c25-4b42-b71c-2bbb20b2a4eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" Nov 02 22:09:04.096806 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.096717 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" Nov 02 22:09:04.103328 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.103303 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sppn6\" (UniqueName: \"kubernetes.io/projected/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-kube-api-access-sppn6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs\" (UID: \"6f310886-2c25-4b42-b71c-2bbb20b2a4eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" Nov 02 22:09:04.198355 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.198319 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" Nov 02 22:09:04.270428 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.270395 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml"] Nov 02 22:09:04.277302 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:09:04.277261 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3e8158_f5c0_4dc2_80b1_257e1ee30487.slice/crio-e52840e6aa56530568d71f050e2938f5b5307b30662d96e3a95448bd95f28a1a WatchSource:0}: Error finding container e52840e6aa56530568d71f050e2938f5b5307b30662d96e3a95448bd95f28a1a: Status 404 returned error can't find the container with id e52840e6aa56530568d71f050e2938f5b5307b30662d96e3a95448bd95f28a1a Nov 02 22:09:04.378654 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.378610 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g"] Nov 02 22:09:04.386653 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:09:04.386617 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a9d920_ed37_4c77_89e6_9c5396dff5e2.slice/crio-50769e929f532826575d982a733ba281907f3a6f685d2fe6428c4e01410577b7 WatchSource:0}: Error finding container 50769e929f532826575d982a733ba281907f3a6f685d2fe6428c4e01410577b7: Status 404 returned error can't find the container with id 50769e929f532826575d982a733ba281907f3a6f685d2fe6428c4e01410577b7 Nov 02 22:09:04.468714 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.468679 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8"] Nov 02 22:09:04.476386 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:09:04.476354 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01be6b4_9954_414a_a080_1a438c467ff3.slice/crio-ca756e99e9c849a5d0bd0234ec13be80e6311fa2f12ce0be69f98eff511e963e WatchSource:0}: Error finding container ca756e99e9c849a5d0bd0234ec13be80e6311fa2f12ce0be69f98eff511e963e: Status 404 returned error can't find the container with id ca756e99e9c849a5d0bd0234ec13be80e6311fa2f12ce0be69f98eff511e963e Nov 02 22:09:04.573752 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.573707 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs"] Nov 02 22:09:04.582696 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:09:04.582667 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f310886_2c25_4b42_b71c_2bbb20b2a4eb.slice/crio-2b865b85898045f6f27035dbf779f0e7b7e3f300ea62fa7c7748a11ce7915024 WatchSource:0}: Error finding container 2b865b85898045f6f27035dbf779f0e7b7e3f300ea62fa7c7748a11ce7915024: Status 404 returned error can't find the container with id 2b865b85898045f6f27035dbf779f0e7b7e3f300ea62fa7c7748a11ce7915024 Nov 02 22:09:04.988053 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.988014 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" event={"ID":"6e3e8158-f5c0-4dc2-80b1-257e1ee30487","Type":"ContainerStarted","Data":"e52840e6aa56530568d71f050e2938f5b5307b30662d96e3a95448bd95f28a1a"} Nov 02 22:09:04.989828 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.989734 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" event={"ID":"6f310886-2c25-4b42-b71c-2bbb20b2a4eb","Type":"ContainerStarted","Data":"2b865b85898045f6f27035dbf779f0e7b7e3f300ea62fa7c7748a11ce7915024"} Nov 02 22:09:04.991443 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.991414 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" event={"ID":"f01be6b4-9954-414a-a080-1a438c467ff3","Type":"ContainerStarted","Data":"ca756e99e9c849a5d0bd0234ec13be80e6311fa2f12ce0be69f98eff511e963e"} Nov 02 22:09:04.993083 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:04.993049 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" event={"ID":"90a9d920-ed37-4c77-89e6-9c5396dff5e2","Type":"ContainerStarted","Data":"50769e929f532826575d982a733ba281907f3a6f685d2fe6428c4e01410577b7"} Nov 02 22:09:06.401845 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:06.401795 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:06.586283 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:06.586231 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:07.007460 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:07.007425 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:07.217445 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:07.217407 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:07.400937 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:07.400831 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:10.881006 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:10.880975 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:11.307460 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.307426 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w"] Nov 02 22:09:11.312144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.312115 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" Nov 02 22:09:11.361886 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.361795 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-lhv59\"" Nov 02 22:09:11.409003 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.408965 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w"] Nov 02 22:09:11.425311 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.425097 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w\" (UID: \"ff95d187-fe3c-44a6-bd58-be88e0cf2aae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" Nov 02 22:09:11.425311 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.425146 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmpd6\" (UniqueName: \"kubernetes.io/projected/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-kube-api-access-tmpd6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w\" (UID: \"ff95d187-fe3c-44a6-bd58-be88e0cf2aae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" Nov 02 22:09:11.425311 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.425182 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w\" (UID: \"ff95d187-fe3c-44a6-bd58-be88e0cf2aae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" Nov 02 22:09:11.525763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.525714 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w\" (UID: \"ff95d187-fe3c-44a6-bd58-be88e0cf2aae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" Nov 02 22:09:11.526056 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.525791 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmpd6\" (UniqueName: \"kubernetes.io/projected/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-kube-api-access-tmpd6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w\" (UID: \"ff95d187-fe3c-44a6-bd58-be88e0cf2aae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" Nov 02 22:09:11.526056 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.525830 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w\" (UID: \"ff95d187-fe3c-44a6-bd58-be88e0cf2aae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" Nov 02 22:09:11.526464 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.526442 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w\" (UID: \"ff95d187-fe3c-44a6-bd58-be88e0cf2aae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" Nov 02 22:09:11.526759 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.526737 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w\" (UID: \"ff95d187-fe3c-44a6-bd58-be88e0cf2aae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" Nov 02 22:09:11.587164 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.587089 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmpd6\" (UniqueName: \"kubernetes.io/projected/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-kube-api-access-tmpd6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w\" (UID: \"ff95d187-fe3c-44a6-bd58-be88e0cf2aae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" Nov 02 22:09:11.634196 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.634147 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" Nov 02 22:09:11.746464 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.746414 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q"] Nov 02 22:09:11.755887 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.755842 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" Nov 02 22:09:11.805612 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.805561 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-x8rmm\"" Nov 02 22:09:11.832383 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.832348 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa4318c2-0167-44ea-bbe9-947f4a85783d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q\" (UID: \"fa4318c2-0167-44ea-bbe9-947f4a85783d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" Nov 02 22:09:11.832605 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.832400 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpskt\" (UniqueName: \"kubernetes.io/projected/fa4318c2-0167-44ea-bbe9-947f4a85783d-kube-api-access-gpskt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q\" (UID: \"fa4318c2-0167-44ea-bbe9-947f4a85783d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" Nov 02 22:09:11.832605 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.832438 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa4318c2-0167-44ea-bbe9-947f4a85783d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q\" (UID: \"fa4318c2-0167-44ea-bbe9-947f4a85783d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" Nov 02 22:09:11.839913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.839794 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q"] Nov 02 22:09:11.933661 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.933621 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa4318c2-0167-44ea-bbe9-947f4a85783d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q\" (UID: \"fa4318c2-0167-44ea-bbe9-947f4a85783d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" Nov 02 22:09:11.934125 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.933668 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpskt\" (UniqueName: \"kubernetes.io/projected/fa4318c2-0167-44ea-bbe9-947f4a85783d-kube-api-access-gpskt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q\" (UID: \"fa4318c2-0167-44ea-bbe9-947f4a85783d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" Nov 02 22:09:11.934125 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.933708 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa4318c2-0167-44ea-bbe9-947f4a85783d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q\" (UID: \"fa4318c2-0167-44ea-bbe9-947f4a85783d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" Nov 02 22:09:11.934285 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.934211 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa4318c2-0167-44ea-bbe9-947f4a85783d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q\" (UID: \"fa4318c2-0167-44ea-bbe9-947f4a85783d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" Nov 02 22:09:11.934805 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.934415 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa4318c2-0167-44ea-bbe9-947f4a85783d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q\" (UID: \"fa4318c2-0167-44ea-bbe9-947f4a85783d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" Nov 02 22:09:11.994854 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:11.994818 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpskt\" (UniqueName: \"kubernetes.io/projected/fa4318c2-0167-44ea-bbe9-947f4a85783d-kube-api-access-gpskt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q\" (UID: \"fa4318c2-0167-44ea-bbe9-947f4a85783d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" Nov 02 22:09:12.050466 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.050428 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w"] Nov 02 22:09:12.059108 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:09:12.059069 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff95d187_fe3c_44a6_bd58_be88e0cf2aae.slice/crio-eff9d69b155f3d0f8fc5e5eecc804cbd598788f15d44272274fcffbb894e7e9a WatchSource:0}: Error finding container eff9d69b155f3d0f8fc5e5eecc804cbd598788f15d44272274fcffbb894e7e9a: Status 404 returned error can't find the container with id eff9d69b155f3d0f8fc5e5eecc804cbd598788f15d44272274fcffbb894e7e9a Nov 02 22:09:12.075293 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.075261 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" Nov 02 22:09:12.496885 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.496787 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q"] Nov 02 22:09:12.511799 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:09:12.511744 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4318c2_0167_44ea_bbe9_947f4a85783d.slice/crio-41b079f6207d6f91265c9937c042924eb17e4b0d26a70c5a4ac3a05911a5c46f WatchSource:0}: Error finding container 41b079f6207d6f91265c9937c042924eb17e4b0d26a70c5a4ac3a05911a5c46f: Status 404 returned error can't find the container with id 41b079f6207d6f91265c9937c042924eb17e4b0d26a70c5a4ac3a05911a5c46f Nov 02 22:09:12.564920 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.564867 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84"] Nov 02 22:09:12.569842 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.569403 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" Nov 02 22:09:12.598454 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.598420 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x"] Nov 02 22:09:12.602512 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.601995 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" Nov 02 22:09:12.625965 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.625931 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84"] Nov 02 22:09:12.632357 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.632316 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb"] Nov 02 22:09:12.637215 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.637188 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" Nov 02 22:09:12.639442 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.639411 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dx29\" (UniqueName: \"kubernetes.io/projected/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-kube-api-access-6dx29\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x\" (UID: \"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" Nov 02 22:09:12.639597 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.639559 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x\" (UID: \"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" Nov 02 22:09:12.639677 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.639595 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b57f0db-4c96-4440-a057-3b51619d9873-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84\" (UID: \"7b57f0db-4c96-4440-a057-3b51619d9873\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" Nov 02 22:09:12.639735 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.639676 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdj46\" (UniqueName: \"kubernetes.io/projected/7b57f0db-4c96-4440-a057-3b51619d9873-kube-api-access-pdj46\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84\" (UID: \"7b57f0db-4c96-4440-a057-3b51619d9873\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" Nov 02 22:09:12.640005 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.639798 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b57f0db-4c96-4440-a057-3b51619d9873-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84\" (UID: \"7b57f0db-4c96-4440-a057-3b51619d9873\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" Nov 02 22:09:12.640005 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.639853 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x\" (UID: \"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" Nov 02 22:09:12.655895 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.655851 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x"] Nov 02 22:09:12.694973 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.694933 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb"] Nov 02 22:09:12.741150 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.741109 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x\" (UID: \"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" Nov 02 22:09:12.741347 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.741164 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b57f0db-4c96-4440-a057-3b51619d9873-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84\" (UID: \"7b57f0db-4c96-4440-a057-3b51619d9873\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" Nov 02 22:09:12.741347 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.741212 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdj46\" (UniqueName: \"kubernetes.io/projected/7b57f0db-4c96-4440-a057-3b51619d9873-kube-api-access-pdj46\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84\" (UID: \"7b57f0db-4c96-4440-a057-3b51619d9873\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" Nov 02 22:09:12.741347 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.741253 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb\" (UID: \"d5009cc4-ddbc-4952-a2c6-6da12f2428e2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" Nov 02 22:09:12.741347 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.741302 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b57f0db-4c96-4440-a057-3b51619d9873-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84\" (UID: \"7b57f0db-4c96-4440-a057-3b51619d9873\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" Nov 02 22:09:12.741347 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.741336 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x\" (UID: \"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" Nov 02 22:09:12.741630 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.741461 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvqm6\" (UniqueName: \"kubernetes.io/projected/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-kube-api-access-vvqm6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb\" (UID: \"d5009cc4-ddbc-4952-a2c6-6da12f2428e2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" Nov 02 22:09:12.741630 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.741529 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb\" (UID: \"d5009cc4-ddbc-4952-a2c6-6da12f2428e2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" Nov 02 22:09:12.741630 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.741581 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dx29\" (UniqueName: \"kubernetes.io/projected/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-kube-api-access-6dx29\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x\" (UID: \"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" Nov 02 22:09:12.741630 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.741599 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x\" (UID: \"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" Nov 02 22:09:12.741630 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.741611 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b57f0db-4c96-4440-a057-3b51619d9873-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84\" (UID: \"7b57f0db-4c96-4440-a057-3b51619d9873\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" Nov 02 22:09:12.741927 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.741766 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b57f0db-4c96-4440-a057-3b51619d9873-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84\" (UID: \"7b57f0db-4c96-4440-a057-3b51619d9873\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" Nov 02 22:09:12.741927 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.741867 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x\" (UID: \"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" Nov 02 22:09:12.800437 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.800359 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdj46\" (UniqueName: \"kubernetes.io/projected/7b57f0db-4c96-4440-a057-3b51619d9873-kube-api-access-pdj46\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84\" (UID: \"7b57f0db-4c96-4440-a057-3b51619d9873\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" Nov 02 22:09:12.801861 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.801821 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dx29\" (UniqueName: \"kubernetes.io/projected/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-kube-api-access-6dx29\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x\" (UID: \"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" Nov 02 22:09:12.842345 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.842298 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb\" (UID: \"d5009cc4-ddbc-4952-a2c6-6da12f2428e2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" Nov 02 22:09:12.842527 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.842372 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb\" (UID: \"d5009cc4-ddbc-4952-a2c6-6da12f2428e2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" Nov 02 22:09:12.842527 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.842432 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvqm6\" (UniqueName: \"kubernetes.io/projected/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-kube-api-access-vvqm6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb\" (UID: \"d5009cc4-ddbc-4952-a2c6-6da12f2428e2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" Nov 02 22:09:12.843073 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.843045 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb\" (UID: \"d5009cc4-ddbc-4952-a2c6-6da12f2428e2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" Nov 02 22:09:12.843311 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.843244 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb\" (UID: \"d5009cc4-ddbc-4952-a2c6-6da12f2428e2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" Nov 02 22:09:12.890071 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.890030 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" Nov 02 22:09:12.918026 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.917985 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvqm6\" (UniqueName: \"kubernetes.io/projected/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-kube-api-access-vvqm6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb\" (UID: \"d5009cc4-ddbc-4952-a2c6-6da12f2428e2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" Nov 02 22:09:12.922603 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.922357 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" Nov 02 22:09:12.960396 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:12.960116 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" Nov 02 22:09:13.050785 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.050657 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" event={"ID":"fa4318c2-0167-44ea-bbe9-947f4a85783d","Type":"ContainerStarted","Data":"41b079f6207d6f91265c9937c042924eb17e4b0d26a70c5a4ac3a05911a5c46f"} Nov 02 22:09:13.052742 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.052663 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" event={"ID":"ff95d187-fe3c-44a6-bd58-be88e0cf2aae","Type":"ContainerStarted","Data":"eff9d69b155f3d0f8fc5e5eecc804cbd598788f15d44272274fcffbb894e7e9a"} Nov 02 22:09:13.197725 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.197684 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx"] Nov 02 22:09:13.201309 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.201281 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" Nov 02 22:09:13.245826 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.245780 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx\" (UID: \"d76ffd5f-9ea1-4165-b735-371ebc0bd59a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" Nov 02 22:09:13.246065 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.245855 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx\" (UID: \"d76ffd5f-9ea1-4165-b735-371ebc0bd59a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" Nov 02 22:09:13.246065 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.245999 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9l68\" (UniqueName: \"kubernetes.io/projected/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-kube-api-access-s9l68\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx\" (UID: \"d76ffd5f-9ea1-4165-b735-371ebc0bd59a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" Nov 02 22:09:13.257021 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.256984 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx"] Nov 02 22:09:13.323701 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.323622 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84"] Nov 02 22:09:13.333028 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:09:13.332990 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b57f0db_4c96_4440_a057_3b51619d9873.slice/crio-edc2c32a05d1820fe233f2296dd08f0f7b5b8966d84f61f5ff239e875e023dbd WatchSource:0}: Error finding container edc2c32a05d1820fe233f2296dd08f0f7b5b8966d84f61f5ff239e875e023dbd: Status 404 returned error can't find the container with id edc2c32a05d1820fe233f2296dd08f0f7b5b8966d84f61f5ff239e875e023dbd Nov 02 22:09:13.347340 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.347301 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx\" (UID: \"d76ffd5f-9ea1-4165-b735-371ebc0bd59a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" Nov 02 22:09:13.348002 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.347359 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx\" (UID: \"d76ffd5f-9ea1-4165-b735-371ebc0bd59a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" Nov 02 22:09:13.348002 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.347387 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9l68\" (UniqueName: \"kubernetes.io/projected/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-kube-api-access-s9l68\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx\" (UID: \"d76ffd5f-9ea1-4165-b735-371ebc0bd59a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" Nov 02 22:09:13.348135 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.348040 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx\" (UID: \"d76ffd5f-9ea1-4165-b735-371ebc0bd59a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" Nov 02 22:09:13.348251 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.348235 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx\" (UID: \"d76ffd5f-9ea1-4165-b735-371ebc0bd59a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" Nov 02 22:09:13.359750 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:09:13.359708 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58fd6d4e_bdb2_4d51_89ff_7b0fef90f0df.slice/crio-e9435f0b6771dada91f82f0b9a2ed2b5d80825386078619a2605a83f7cddf828 WatchSource:0}: Error finding container e9435f0b6771dada91f82f0b9a2ed2b5d80825386078619a2605a83f7cddf828: Status 404 returned error can't find the container with id e9435f0b6771dada91f82f0b9a2ed2b5d80825386078619a2605a83f7cddf828 Nov 02 22:09:13.362334 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.362237 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x"] Nov 02 22:09:13.401139 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.401084 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb"] Nov 02 22:09:13.413346 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:09:13.413300 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5009cc4_ddbc_4952_a2c6_6da12f2428e2.slice/crio-fb2b4e123062e32f9d5c7e2bd1f25330c5fc6829c6a9e2b1ff85a9e4617a7cfa WatchSource:0}: Error finding container fb2b4e123062e32f9d5c7e2bd1f25330c5fc6829c6a9e2b1ff85a9e4617a7cfa: Status 404 returned error can't find the container with id fb2b4e123062e32f9d5c7e2bd1f25330c5fc6829c6a9e2b1ff85a9e4617a7cfa Nov 02 22:09:13.414829 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.414800 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9l68\" (UniqueName: \"kubernetes.io/projected/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-kube-api-access-s9l68\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx\" (UID: \"d76ffd5f-9ea1-4165-b735-371ebc0bd59a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" Nov 02 22:09:13.522487 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.522441 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" Nov 02 22:09:13.802578 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:09:13.802540 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:09:13.940084 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:13.939922 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx"] Nov 02 22:09:13.950425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:09:13.950378 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76ffd5f_9ea1_4165_b735_371ebc0bd59a.slice/crio-1bc15d57bec61020fb730ac8eb9dd1644293c1a89cb3ad60e58d198117c8229e WatchSource:0}: Error finding container 1bc15d57bec61020fb730ac8eb9dd1644293c1a89cb3ad60e58d198117c8229e: Status 404 returned error can't find the container with id 1bc15d57bec61020fb730ac8eb9dd1644293c1a89cb3ad60e58d198117c8229e Nov 02 22:09:14.060914 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:14.060814 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" event={"ID":"d76ffd5f-9ea1-4165-b735-371ebc0bd59a","Type":"ContainerStarted","Data":"1bc15d57bec61020fb730ac8eb9dd1644293c1a89cb3ad60e58d198117c8229e"} Nov 02 22:09:14.062601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:14.062527 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" event={"ID":"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df","Type":"ContainerStarted","Data":"e9435f0b6771dada91f82f0b9a2ed2b5d80825386078619a2605a83f7cddf828"} Nov 02 22:09:14.064591 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:14.064561 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" event={"ID":"d5009cc4-ddbc-4952-a2c6-6da12f2428e2","Type":"ContainerStarted","Data":"fb2b4e123062e32f9d5c7e2bd1f25330c5fc6829c6a9e2b1ff85a9e4617a7cfa"} Nov 02 22:09:14.067599 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:14.067570 5871 generic.go:356] "Generic (PLEG): container finished" podID="fa4318c2-0167-44ea-bbe9-947f4a85783d" containerID="c7cfc8e5463cf8f44874afcf449de8ce14dfd1c72c8a486c5101ae014acf6c20" exitCode=0 Nov 02 22:09:14.067734 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:14.067662 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" event={"ID":"fa4318c2-0167-44ea-bbe9-947f4a85783d","Type":"ContainerDied","Data":"c7cfc8e5463cf8f44874afcf449de8ce14dfd1c72c8a486c5101ae014acf6c20"} Nov 02 22:09:14.070097 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:14.069983 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" event={"ID":"7b57f0db-4c96-4440-a057-3b51619d9873","Type":"ContainerStarted","Data":"edc2c32a05d1820fe233f2296dd08f0f7b5b8966d84f61f5ff239e875e023dbd"} Nov 02 22:09:14.072850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:14.072100 5871 generic.go:356] "Generic (PLEG): container finished" podID="0174771e-0c4e-4d4d-9f5f-a260a6719f1a" containerID="9226f8a47ddc645105d6f66877aaf11896e1d3b839d5211c677ab9b6946e844d" exitCode=0 Nov 02 22:09:14.072850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:14.072158 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" event={"ID":"0174771e-0c4e-4d4d-9f5f-a260a6719f1a","Type":"ContainerDied","Data":"9226f8a47ddc645105d6f66877aaf11896e1d3b839d5211c677ab9b6946e844d"} Nov 02 22:09:15.080801 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:15.080752 5871 generic.go:356] "Generic (PLEG): container finished" podID="d5009cc4-ddbc-4952-a2c6-6da12f2428e2" containerID="8161868d49e714058fba9c044aef4c2482e8d3f47d5784e0a45668fd183c43c0" exitCode=0 Nov 02 22:09:15.081428 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:15.080909 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" event={"ID":"d5009cc4-ddbc-4952-a2c6-6da12f2428e2","Type":"ContainerDied","Data":"8161868d49e714058fba9c044aef4c2482e8d3f47d5784e0a45668fd183c43c0"} Nov 02 22:09:15.085331 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:15.085305 5871 generic.go:356] "Generic (PLEG): container finished" podID="7b57f0db-4c96-4440-a057-3b51619d9873" containerID="ef8d3ef98f3fd9ba67edb42491e8f9660a5daeb1935235db9638dedca6c54d20" exitCode=0 Nov 02 22:09:15.085453 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:15.085430 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" event={"ID":"7b57f0db-4c96-4440-a057-3b51619d9873","Type":"ContainerDied","Data":"ef8d3ef98f3fd9ba67edb42491e8f9660a5daeb1935235db9638dedca6c54d20"} Nov 02 22:09:15.088100 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:15.088072 5871 generic.go:356] "Generic (PLEG): container finished" podID="d76ffd5f-9ea1-4165-b735-371ebc0bd59a" containerID="1fd655b2efa8f30ff767eb148384e2b598187fe2a0735a7d2311ea94ba7be762" exitCode=0 Nov 02 22:09:15.088207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:15.088173 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" event={"ID":"d76ffd5f-9ea1-4165-b735-371ebc0bd59a","Type":"ContainerDied","Data":"1fd655b2efa8f30ff767eb148384e2b598187fe2a0735a7d2311ea94ba7be762"} Nov 02 22:09:15.092291 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:15.092266 5871 generic.go:356] "Generic (PLEG): container finished" podID="58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df" containerID="35ce17c0fa51a6edea65117f79dbf22a1dc3a7b43928b2cd4f96e1239ae079d1" exitCode=0 Nov 02 22:09:15.092423 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:15.092312 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" event={"ID":"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df","Type":"ContainerDied","Data":"35ce17c0fa51a6edea65117f79dbf22a1dc3a7b43928b2cd4f96e1239ae079d1"} Nov 02 22:09:17.110626 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:17.110543 5871 generic.go:356] "Generic (PLEG): container finished" podID="6f310886-2c25-4b42-b71c-2bbb20b2a4eb" containerID="86c3d84734438b28d3aeca3815d065e56ec935726a66438cbbc28b08138771f3" exitCode=0 Nov 02 22:09:17.111045 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:17.110638 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" event={"ID":"6f310886-2c25-4b42-b71c-2bbb20b2a4eb","Type":"ContainerDied","Data":"86c3d84734438b28d3aeca3815d065e56ec935726a66438cbbc28b08138771f3"} Nov 02 22:09:17.114675 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:17.114646 5871 generic.go:356] "Generic (PLEG): container finished" podID="f01be6b4-9954-414a-a080-1a438c467ff3" containerID="52c19c5c8a890a5b3b4ef0c60bf1cf02be0ddea0b8cff668688f1184d35b6453" exitCode=0 Nov 02 22:09:17.114788 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:17.114752 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" event={"ID":"f01be6b4-9954-414a-a080-1a438c467ff3","Type":"ContainerDied","Data":"52c19c5c8a890a5b3b4ef0c60bf1cf02be0ddea0b8cff668688f1184d35b6453"} Nov 02 22:09:17.119808 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:17.117988 5871 generic.go:356] "Generic (PLEG): container finished" podID="90a9d920-ed37-4c77-89e6-9c5396dff5e2" containerID="79c7dcab401e6179418ef17ebf6060bd59fa9e522c7c1bc1c81ea9cb5dedbd17" exitCode=0 Nov 02 22:09:17.119808 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:17.118116 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" event={"ID":"90a9d920-ed37-4c77-89e6-9c5396dff5e2","Type":"ContainerDied","Data":"79c7dcab401e6179418ef17ebf6060bd59fa9e522c7c1bc1c81ea9cb5dedbd17"} Nov 02 22:09:17.124004 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:17.123307 5871 generic.go:356] "Generic (PLEG): container finished" podID="6e3e8158-f5c0-4dc2-80b1-257e1ee30487" containerID="bee5b3d4ee59cbadbc5eafcda3fd525e458685e33795fe1e9ca182886eb53400" exitCode=0 Nov 02 22:09:17.124004 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:17.123360 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" event={"ID":"6e3e8158-f5c0-4dc2-80b1-257e1ee30487","Type":"ContainerDied","Data":"bee5b3d4ee59cbadbc5eafcda3fd525e458685e33795fe1e9ca182886eb53400"} Nov 02 22:09:18.132359 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:18.132268 5871 generic.go:356] "Generic (PLEG): container finished" podID="58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df" containerID="dfa4c351aad49dfdaaf479f83157fd754bcc18e54f2c6203ace5ef2f50423f19" exitCode=0 Nov 02 22:09:18.132722 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:18.132378 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" event={"ID":"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df","Type":"ContainerDied","Data":"dfa4c351aad49dfdaaf479f83157fd754bcc18e54f2c6203ace5ef2f50423f19"} Nov 02 22:09:18.139625 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:18.139593 5871 generic.go:356] "Generic (PLEG): container finished" podID="d5009cc4-ddbc-4952-a2c6-6da12f2428e2" containerID="193f3456f6ba9a6328b3d108185f0245a999a23d082a8a03af2dd2e3edbeda43" exitCode=0 Nov 02 22:09:18.140417 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:18.139786 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" event={"ID":"d5009cc4-ddbc-4952-a2c6-6da12f2428e2","Type":"ContainerDied","Data":"193f3456f6ba9a6328b3d108185f0245a999a23d082a8a03af2dd2e3edbeda43"} Nov 02 22:09:18.143044 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:18.142178 5871 generic.go:356] "Generic (PLEG): container finished" podID="7b57f0db-4c96-4440-a057-3b51619d9873" containerID="f61bcf56517d12699a74a4f4d8a9ec50ecd55bda52a4f66ea19d8bb69927e035" exitCode=0 Nov 02 22:09:18.143044 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:18.142263 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" event={"ID":"7b57f0db-4c96-4440-a057-3b51619d9873","Type":"ContainerDied","Data":"f61bcf56517d12699a74a4f4d8a9ec50ecd55bda52a4f66ea19d8bb69927e035"} Nov 02 22:09:18.851224 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:18.851190 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:19.150738 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:19.150704 5871 generic.go:356] "Generic (PLEG): container finished" podID="6e3e8158-f5c0-4dc2-80b1-257e1ee30487" containerID="3881266f6f6eb07872b3efb01d00c225cac9097c9c1e023bc8272e0eae811e94" exitCode=0 Nov 02 22:09:19.151138 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:19.150800 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" event={"ID":"6e3e8158-f5c0-4dc2-80b1-257e1ee30487","Type":"ContainerDied","Data":"3881266f6f6eb07872b3efb01d00c225cac9097c9c1e023bc8272e0eae811e94"} Nov 02 22:09:19.155605 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:19.155573 5871 generic.go:356] "Generic (PLEG): container finished" podID="0174771e-0c4e-4d4d-9f5f-a260a6719f1a" containerID="0271b1bb7e8efd10fc65ea076e279f99ff3951cab1d59a8e86b3b4bfd8e96e40" exitCode=0 Nov 02 22:09:19.155716 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:19.155675 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" event={"ID":"0174771e-0c4e-4d4d-9f5f-a260a6719f1a","Type":"ContainerDied","Data":"0271b1bb7e8efd10fc65ea076e279f99ff3951cab1d59a8e86b3b4bfd8e96e40"} Nov 02 22:09:19.161269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:19.160415 5871 generic.go:356] "Generic (PLEG): container finished" podID="6f310886-2c25-4b42-b71c-2bbb20b2a4eb" containerID="f52bb4217f88d02463dd9fc4dade6a3af3120dbbbfcc93ce1099791df3553453" exitCode=0 Nov 02 22:09:19.161269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:19.160789 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" event={"ID":"6f310886-2c25-4b42-b71c-2bbb20b2a4eb","Type":"ContainerDied","Data":"f52bb4217f88d02463dd9fc4dade6a3af3120dbbbfcc93ce1099791df3553453"} Nov 02 22:09:19.171406 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:19.171372 5871 generic.go:356] "Generic (PLEG): container finished" podID="f01be6b4-9954-414a-a080-1a438c467ff3" containerID="fb28bd160f731573a0947ee9952017575521c4f4c117938adcb3617d6ba49727" exitCode=0 Nov 02 22:09:19.171557 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:19.171484 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" event={"ID":"f01be6b4-9954-414a-a080-1a438c467ff3","Type":"ContainerDied","Data":"fb28bd160f731573a0947ee9952017575521c4f4c117938adcb3617d6ba49727"} Nov 02 22:09:20.184449 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:20.184316 5871 generic.go:356] "Generic (PLEG): container finished" podID="fa4318c2-0167-44ea-bbe9-947f4a85783d" containerID="066695bbd35225773075e0cd5dd67d79e2f71a733d9ecc0acf2d68cd14da83c2" exitCode=0 Nov 02 22:09:20.184942 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:20.184447 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" event={"ID":"fa4318c2-0167-44ea-bbe9-947f4a85783d","Type":"ContainerDied","Data":"066695bbd35225773075e0cd5dd67d79e2f71a733d9ecc0acf2d68cd14da83c2"} Nov 02 22:09:20.195670 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:20.195637 5871 generic.go:356] "Generic (PLEG): container finished" podID="ff95d187-fe3c-44a6-bd58-be88e0cf2aae" containerID="9ee14ee8ad499dc942eb75edc73bd44c043fc72df0c76bf0edf900d41305265f" exitCode=0 Nov 02 22:09:20.196057 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:20.196031 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" event={"ID":"ff95d187-fe3c-44a6-bd58-be88e0cf2aae","Type":"ContainerDied","Data":"9ee14ee8ad499dc942eb75edc73bd44c043fc72df0c76bf0edf900d41305265f"} Nov 02 22:09:20.202658 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:20.201773 5871 generic.go:356] "Generic (PLEG): container finished" podID="d76ffd5f-9ea1-4165-b735-371ebc0bd59a" containerID="447b4f9e51cde42e88586c3d0ed66496a8d530268069e5a2453b3916a88edfa5" exitCode=0 Nov 02 22:09:20.202658 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:20.201886 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" event={"ID":"d76ffd5f-9ea1-4165-b735-371ebc0bd59a","Type":"ContainerDied","Data":"447b4f9e51cde42e88586c3d0ed66496a8d530268069e5a2453b3916a88edfa5"} Nov 02 22:09:20.211115 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:20.211046 5871 generic.go:356] "Generic (PLEG): container finished" podID="90a9d920-ed37-4c77-89e6-9c5396dff5e2" containerID="562471b5297a500f2cd868e803013b53e73fcbcd6c3e6dee0e9b9e7f2f075cfe" exitCode=0 Nov 02 22:09:20.211225 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:20.211171 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" event={"ID":"90a9d920-ed37-4c77-89e6-9c5396dff5e2","Type":"ContainerDied","Data":"562471b5297a500f2cd868e803013b53e73fcbcd6c3e6dee0e9b9e7f2f075cfe"} Nov 02 22:09:22.229130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:22.228124 5871 generic.go:356] "Generic (PLEG): container finished" podID="ff95d187-fe3c-44a6-bd58-be88e0cf2aae" containerID="7ba8694fe9b895e19baa8e4cfd5be4d65d6131a1bc262c40dd0e5cf8ae48abe1" exitCode=0 Nov 02 22:09:22.229130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:22.228194 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" event={"ID":"ff95d187-fe3c-44a6-bd58-be88e0cf2aae","Type":"ContainerDied","Data":"7ba8694fe9b895e19baa8e4cfd5be4d65d6131a1bc262c40dd0e5cf8ae48abe1"} Nov 02 22:09:23.490238 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:23.490198 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:35.638142 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:35.638042 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:35.814595 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:35.814518 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:41.465397 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:41.464775 5871 generic.go:356] "Generic (PLEG): container finished" podID="58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df" containerID="d5ca178c27b3d032d8added7b52fbc41d98610dfb6e6fd8081f6413e9a5f6e6d" exitCode=0 Nov 02 22:09:41.465397 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:41.464929 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" event={"ID":"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df","Type":"ContainerDied","Data":"d5ca178c27b3d032d8added7b52fbc41d98610dfb6e6fd8081f6413e9a5f6e6d"} Nov 02 22:09:41.469953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:41.469916 5871 generic.go:356] "Generic (PLEG): container finished" podID="d5009cc4-ddbc-4952-a2c6-6da12f2428e2" containerID="d84d64523d6b0d5e55af6351b14c8b2701f3e06da601c842c261542481e37298" exitCode=0 Nov 02 22:09:41.470095 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:41.469961 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" event={"ID":"d5009cc4-ddbc-4952-a2c6-6da12f2428e2","Type":"ContainerDied","Data":"d84d64523d6b0d5e55af6351b14c8b2701f3e06da601c842c261542481e37298"} Nov 02 22:09:42.481583 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.481487 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" event={"ID":"6e3e8158-f5c0-4dc2-80b1-257e1ee30487","Type":"ContainerStarted","Data":"21863be1b1012bcef56ed9a75bfcaf91765cdaa359fdd4c7642622035d049827"} Nov 02 22:09:42.484864 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.484829 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" event={"ID":"7b57f0db-4c96-4440-a057-3b51619d9873","Type":"ContainerStarted","Data":"feb8c373f37753e7d35929eed6b27945acc784ba31bca582da817346d6453b1a"} Nov 02 22:09:42.497120 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.497084 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" event={"ID":"0174771e-0c4e-4d4d-9f5f-a260a6719f1a","Type":"ContainerStarted","Data":"0f955c04e76b3188eaf3419f798028fb26d69d5ad55f9133e55a0e3d8d93bb58"} Nov 02 22:09:42.593517 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.592603 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" podStartSLOduration=2.310757305 podStartE2EDuration="39.592587779s" totalImagesPullingTime="37.281830474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:42.591703394 +0000 UTC m=+205.404033574" watchObservedRunningTime="2026-11-02 22:09:42.592587779 +0000 UTC m=+205.404917950" Nov 02 22:09:42.693076 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.693029 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" Nov 02 22:09:42.731361 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.731267 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" Nov 02 22:09:42.773123 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.773088 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-util\" (UniqueName: \"kubernetes.io/empty-dir/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-util\") pod \"d5009cc4-ddbc-4952-a2c6-6da12f2428e2\" (UID: \"d5009cc4-ddbc-4952-a2c6-6da12f2428e2\") " Nov 02 22:09:42.773304 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.773146 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-bundle\") pod \"d5009cc4-ddbc-4952-a2c6-6da12f2428e2\" (UID: \"d5009cc4-ddbc-4952-a2c6-6da12f2428e2\") " Nov 02 22:09:42.773304 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.773168 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-kube-api-access-vvqm6\" (UniqueName: \"kubernetes.io/projected/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-kube-api-access-vvqm6\") pod \"d5009cc4-ddbc-4952-a2c6-6da12f2428e2\" (UID: \"d5009cc4-ddbc-4952-a2c6-6da12f2428e2\") " Nov 02 22:09:42.773304 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.773191 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-bundle\" (UniqueName: \"kubernetes.io/empty-dir/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-bundle\") pod \"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df\" (UID: \"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df\") " Nov 02 22:09:42.773304 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.773255 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-kube-api-access-6dx29\" (UniqueName: \"kubernetes.io/projected/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-kube-api-access-6dx29\") pod \"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df\" (UID: \"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df\") " Nov 02 22:09:42.773304 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.773283 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-util\" (UniqueName: \"kubernetes.io/empty-dir/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-util\") pod \"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df\" (UID: \"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df\") " Nov 02 22:09:42.774317 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.774270 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-bundle" pod "58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df" (UID: "58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:42.775247 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.775217 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-bundle" pod "d5009cc4-ddbc-4952-a2c6-6da12f2428e2" (UID: "d5009cc4-ddbc-4952-a2c6-6da12f2428e2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:42.780443 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.780403 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-util" pod "d5009cc4-ddbc-4952-a2c6-6da12f2428e2" (UID: "d5009cc4-ddbc-4952-a2c6-6da12f2428e2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:42.781814 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.781713 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-util" pod "58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df" (UID: "58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:42.784407 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.784377 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-kube-api-access-vvqm6" pod "d5009cc4-ddbc-4952-a2c6-6da12f2428e2" (UID: "d5009cc4-ddbc-4952-a2c6-6da12f2428e2"). InnerVolumeSpecName "kube-api-access-vvqm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:42.784986 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.784843 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-kube-api-access-6dx29" pod "58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df" (UID: "58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df"). InnerVolumeSpecName "kube-api-access-6dx29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:42.819390 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.818734 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" podStartSLOduration=3.211127953 podStartE2EDuration="40.818716395s" totalImagesPullingTime="37.607588442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:42.7079583 +0000 UTC m=+205.520288482" watchObservedRunningTime="2026-11-02 22:09:42.818716395 +0000 UTC m=+205.631046569" Nov 02 22:09:42.820496 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.820441 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" podStartSLOduration=3.273664431 podStartE2EDuration="30.820430154s" totalImagesPullingTime="27.546765723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:42.818317181 +0000 UTC m=+205.630647362" watchObservedRunningTime="2026-11-02 22:09:42.820430154 +0000 UTC m=+205.632760333" Nov 02 22:09:42.874917 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.874850 5871 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-util\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:42.874917 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.874901 5871 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:42.874917 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.874917 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvqm6\" (UniqueName: \"kubernetes.io/projected/d5009cc4-ddbc-4952-a2c6-6da12f2428e2-kube-api-access-vvqm6\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:42.875205 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.874932 5871 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:42.875205 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.874947 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dx29\" (UniqueName: \"kubernetes.io/projected/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-kube-api-access-6dx29\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:42.875205 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:42.874961 5871 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df-util\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:43.504538 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:43.504502 5871 generic.go:356] "Generic (PLEG): container finished" podID="6e3e8158-f5c0-4dc2-80b1-257e1ee30487" containerID="21863be1b1012bcef56ed9a75bfcaf91765cdaa359fdd4c7642622035d049827" exitCode=0 Nov 02 22:09:43.505060 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:43.504613 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" event={"ID":"6e3e8158-f5c0-4dc2-80b1-257e1ee30487","Type":"ContainerDied","Data":"21863be1b1012bcef56ed9a75bfcaf91765cdaa359fdd4c7642622035d049827"} Nov 02 22:09:43.506843 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:43.506812 5871 generic.go:356] "Generic (PLEG): container finished" podID="7b57f0db-4c96-4440-a057-3b51619d9873" containerID="feb8c373f37753e7d35929eed6b27945acc784ba31bca582da817346d6453b1a" exitCode=0 Nov 02 22:09:43.506992 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:43.506900 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" event={"ID":"7b57f0db-4c96-4440-a057-3b51619d9873","Type":"ContainerDied","Data":"feb8c373f37753e7d35929eed6b27945acc784ba31bca582da817346d6453b1a"} Nov 02 22:09:43.510355 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:43.510330 5871 generic.go:356] "Generic (PLEG): container finished" podID="0174771e-0c4e-4d4d-9f5f-a260a6719f1a" containerID="0f955c04e76b3188eaf3419f798028fb26d69d5ad55f9133e55a0e3d8d93bb58" exitCode=0 Nov 02 22:09:43.510464 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:43.510421 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" event={"ID":"0174771e-0c4e-4d4d-9f5f-a260a6719f1a","Type":"ContainerDied","Data":"0f955c04e76b3188eaf3419f798028fb26d69d5ad55f9133e55a0e3d8d93bb58"} Nov 02 22:09:43.512808 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:43.512785 5871 generic.go:356] "Generic (PLEG): container finished" podID="6f310886-2c25-4b42-b71c-2bbb20b2a4eb" containerID="c328e12573afa716637bf516e51c07d942123924237f3c18a2bfb8a17973d40c" exitCode=0 Nov 02 22:09:43.512933 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:43.512914 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" event={"ID":"6f310886-2c25-4b42-b71c-2bbb20b2a4eb","Type":"ContainerDied","Data":"c328e12573afa716637bf516e51c07d942123924237f3c18a2bfb8a17973d40c"} Nov 02 22:09:43.515970 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:43.515566 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" event={"ID":"f01be6b4-9954-414a-a080-1a438c467ff3","Type":"ContainerStarted","Data":"b23836c740532a9e798d70bd5bf4902a769992cc2357f28033eb91e534af8425"} Nov 02 22:09:43.517797 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:43.517774 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" event={"ID":"58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df","Type":"ContainerDied","Data":"e9435f0b6771dada91f82f0b9a2ed2b5d80825386078619a2605a83f7cddf828"} Nov 02 22:09:43.517926 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:43.517800 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cm42x" Nov 02 22:09:43.517926 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:43.517815 5871 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9435f0b6771dada91f82f0b9a2ed2b5d80825386078619a2605a83f7cddf828" Nov 02 22:09:43.520511 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:43.520079 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" event={"ID":"d5009cc4-ddbc-4952-a2c6-6da12f2428e2","Type":"ContainerDied","Data":"fb2b4e123062e32f9d5c7e2bd1f25330c5fc6829c6a9e2b1ff85a9e4617a7cfa"} Nov 02 22:09:43.520511 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:43.520116 5871 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb2b4e123062e32f9d5c7e2bd1f25330c5fc6829c6a9e2b1ff85a9e4617a7cfa" Nov 02 22:09:43.520511 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:43.520178 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2kbtb" Nov 02 22:09:43.773222 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:09:43.773131 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:09:44.154428 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.154373 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" podStartSLOduration=3.714809986 podStartE2EDuration="41.154357819s" totalImagesPullingTime="37.439547833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:44.152739638 +0000 UTC m=+206.965069815" watchObservedRunningTime="2026-11-02 22:09:44.154357819 +0000 UTC m=+206.966687997" Nov 02 22:09:44.530250 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.529694 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" event={"ID":"fa4318c2-0167-44ea-bbe9-947f4a85783d","Type":"ContainerDied","Data":"a4f1e046e324689def394474fd3d4cdd23a63f250f987f40df9d8581a6a591f8"} Nov 02 22:09:44.530631 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.529414 5871 generic.go:356] "Generic (PLEG): container finished" podID="fa4318c2-0167-44ea-bbe9-947f4a85783d" containerID="a4f1e046e324689def394474fd3d4cdd23a63f250f987f40df9d8581a6a591f8" exitCode=0 Nov 02 22:09:44.532663 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.532635 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" event={"ID":"ff95d187-fe3c-44a6-bd58-be88e0cf2aae","Type":"ContainerStarted","Data":"8905ab3dd592b9c97e0013fa57ae1a6a6d851c6eab6353ed57e67b466f64c776"} Nov 02 22:09:44.534641 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.534616 5871 generic.go:356] "Generic (PLEG): container finished" podID="f01be6b4-9954-414a-a080-1a438c467ff3" containerID="b23836c740532a9e798d70bd5bf4902a769992cc2357f28033eb91e534af8425" exitCode=0 Nov 02 22:09:44.534758 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.534699 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" event={"ID":"f01be6b4-9954-414a-a080-1a438c467ff3","Type":"ContainerDied","Data":"b23836c740532a9e798d70bd5bf4902a769992cc2357f28033eb91e534af8425"} Nov 02 22:09:44.536670 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.536650 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" event={"ID":"d76ffd5f-9ea1-4165-b735-371ebc0bd59a","Type":"ContainerStarted","Data":"82b4d2201af1f9404a277cf16b5ff2e3b02b8b3a2924281cef799c511f9babb0"} Nov 02 22:09:44.538708 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.538686 5871 generic.go:356] "Generic (PLEG): container finished" podID="90a9d920-ed37-4c77-89e6-9c5396dff5e2" containerID="50278735adb4ca5f90034240fc617893cd2d2b9680d62837ea2dbc7dbce1743a" exitCode=0 Nov 02 22:09:44.538825 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.538765 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" event={"ID":"90a9d920-ed37-4c77-89e6-9c5396dff5e2","Type":"ContainerDied","Data":"50278735adb4ca5f90034240fc617893cd2d2b9680d62837ea2dbc7dbce1743a"} Nov 02 22:09:44.688743 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.688707 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" Nov 02 22:09:44.739008 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.721991 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" Nov 02 22:09:44.791080 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.791045 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b57f0db-4c96-4440-a057-3b51619d9873-util\" (UniqueName: \"kubernetes.io/empty-dir/7b57f0db-4c96-4440-a057-3b51619d9873-util\") pod \"7b57f0db-4c96-4440-a057-3b51619d9873\" (UID: \"7b57f0db-4c96-4440-a057-3b51619d9873\") " Nov 02 22:09:44.791264 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.791105 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-kube-api-access-grqr9\" (UniqueName: \"kubernetes.io/projected/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-kube-api-access-grqr9\") pod \"6e3e8158-f5c0-4dc2-80b1-257e1ee30487\" (UID: \"6e3e8158-f5c0-4dc2-80b1-257e1ee30487\") " Nov 02 22:09:44.791264 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.791155 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b57f0db-4c96-4440-a057-3b51619d9873-kube-api-access-pdj46\" (UniqueName: \"kubernetes.io/projected/7b57f0db-4c96-4440-a057-3b51619d9873-kube-api-access-pdj46\") pod \"7b57f0db-4c96-4440-a057-3b51619d9873\" (UID: \"7b57f0db-4c96-4440-a057-3b51619d9873\") " Nov 02 22:09:44.791264 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.791189 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-bundle\") pod \"6e3e8158-f5c0-4dc2-80b1-257e1ee30487\" (UID: \"6e3e8158-f5c0-4dc2-80b1-257e1ee30487\") " Nov 02 22:09:44.791264 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.791230 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-util\" (UniqueName: \"kubernetes.io/empty-dir/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-util\") pod \"6e3e8158-f5c0-4dc2-80b1-257e1ee30487\" (UID: \"6e3e8158-f5c0-4dc2-80b1-257e1ee30487\") " Nov 02 22:09:44.791537 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.791277 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b57f0db-4c96-4440-a057-3b51619d9873-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b57f0db-4c96-4440-a057-3b51619d9873-bundle\") pod \"7b57f0db-4c96-4440-a057-3b51619d9873\" (UID: \"7b57f0db-4c96-4440-a057-3b51619d9873\") " Nov 02 22:09:44.792088 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.792051 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b57f0db-4c96-4440-a057-3b51619d9873-bundle" pod "7b57f0db-4c96-4440-a057-3b51619d9873" (UID: "7b57f0db-4c96-4440-a057-3b51619d9873"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:44.792189 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.792078 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-bundle" pod "6e3e8158-f5c0-4dc2-80b1-257e1ee30487" (UID: "6e3e8158-f5c0-4dc2-80b1-257e1ee30487"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:44.793438 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.793405 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-kube-api-access-grqr9" pod "6e3e8158-f5c0-4dc2-80b1-257e1ee30487" (UID: "6e3e8158-f5c0-4dc2-80b1-257e1ee30487"). InnerVolumeSpecName "kube-api-access-grqr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:44.793438 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.793414 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b57f0db-4c96-4440-a057-3b51619d9873-kube-api-access-pdj46" pod "7b57f0db-4c96-4440-a057-3b51619d9873" (UID: "7b57f0db-4c96-4440-a057-3b51619d9873"). InnerVolumeSpecName "kube-api-access-pdj46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:44.795324 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.795303 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b57f0db-4c96-4440-a057-3b51619d9873-util" pod "7b57f0db-4c96-4440-a057-3b51619d9873" (UID: "7b57f0db-4c96-4440-a057-3b51619d9873"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:44.797232 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.797202 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-util" pod "6e3e8158-f5c0-4dc2-80b1-257e1ee30487" (UID: "6e3e8158-f5c0-4dc2-80b1-257e1ee30487"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:44.830416 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.830388 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" Nov 02 22:09:44.832519 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.832496 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" Nov 02 22:09:44.892040 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.892005 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-util\" (UniqueName: \"kubernetes.io/empty-dir/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-util\") pod \"6f310886-2c25-4b42-b71c-2bbb20b2a4eb\" (UID: \"6f310886-2c25-4b42-b71c-2bbb20b2a4eb\") " Nov 02 22:09:44.892227 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.892080 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-kube-api-access-sppn6\" (UniqueName: \"kubernetes.io/projected/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-kube-api-access-sppn6\") pod \"6f310886-2c25-4b42-b71c-2bbb20b2a4eb\" (UID: \"6f310886-2c25-4b42-b71c-2bbb20b2a4eb\") " Nov 02 22:09:44.892227 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.892116 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-util\" (UniqueName: \"kubernetes.io/empty-dir/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-util\") pod \"0174771e-0c4e-4d4d-9f5f-a260a6719f1a\" (UID: \"0174771e-0c4e-4d4d-9f5f-a260a6719f1a\") " Nov 02 22:09:44.892227 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.892140 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-kube-api-access-5n2sd\" (UniqueName: \"kubernetes.io/projected/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-kube-api-access-5n2sd\") pod \"0174771e-0c4e-4d4d-9f5f-a260a6719f1a\" (UID: \"0174771e-0c4e-4d4d-9f5f-a260a6719f1a\") " Nov 02 22:09:44.892227 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.892186 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-bundle\") pod \"0174771e-0c4e-4d4d-9f5f-a260a6719f1a\" (UID: \"0174771e-0c4e-4d4d-9f5f-a260a6719f1a\") " Nov 02 22:09:44.892227 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.892215 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-bundle\") pod \"6f310886-2c25-4b42-b71c-2bbb20b2a4eb\" (UID: \"6f310886-2c25-4b42-b71c-2bbb20b2a4eb\") " Nov 02 22:09:44.892507 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.892457 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grqr9\" (UniqueName: \"kubernetes.io/projected/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-kube-api-access-grqr9\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:44.892507 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.892482 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdj46\" (UniqueName: \"kubernetes.io/projected/7b57f0db-4c96-4440-a057-3b51619d9873-kube-api-access-pdj46\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:44.892507 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.892497 5871 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:44.892667 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.892511 5871 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e3e8158-f5c0-4dc2-80b1-257e1ee30487-util\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:44.892667 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.892524 5871 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b57f0db-4c96-4440-a057-3b51619d9873-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:44.892667 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.892536 5871 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b57f0db-4c96-4440-a057-3b51619d9873-util\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:44.892822 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.892744 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-bundle" pod "6f310886-2c25-4b42-b71c-2bbb20b2a4eb" (UID: "6f310886-2c25-4b42-b71c-2bbb20b2a4eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:44.893228 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.893186 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-bundle" pod "0174771e-0c4e-4d4d-9f5f-a260a6719f1a" (UID: "0174771e-0c4e-4d4d-9f5f-a260a6719f1a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:44.894803 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.894773 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-kube-api-access-sppn6" pod "6f310886-2c25-4b42-b71c-2bbb20b2a4eb" (UID: "6f310886-2c25-4b42-b71c-2bbb20b2a4eb"). InnerVolumeSpecName "kube-api-access-sppn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:44.894966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.894947 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-kube-api-access-5n2sd" pod "0174771e-0c4e-4d4d-9f5f-a260a6719f1a" (UID: "0174771e-0c4e-4d4d-9f5f-a260a6719f1a"). InnerVolumeSpecName "kube-api-access-5n2sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:44.898057 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.898032 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-util" pod "0174771e-0c4e-4d4d-9f5f-a260a6719f1a" (UID: "0174771e-0c4e-4d4d-9f5f-a260a6719f1a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:44.899195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.899172 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-util" pod "6f310886-2c25-4b42-b71c-2bbb20b2a4eb" (UID: "6f310886-2c25-4b42-b71c-2bbb20b2a4eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:44.992763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.992630 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" podStartSLOduration=3.6108710410000002 podStartE2EDuration="33.992610879s" totalImagesPullingTime="30.381739838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:44.99208475 +0000 UTC m=+207.804414930" watchObservedRunningTime="2026-11-02 22:09:44.992610879 +0000 UTC m=+207.804941059" Nov 02 22:09:44.993457 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.993434 5871 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-util\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:44.993549 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.993460 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sppn6\" (UniqueName: \"kubernetes.io/projected/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-kube-api-access-sppn6\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:44.993549 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.993472 5871 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-util\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:44.993549 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.993483 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n2sd\" (UniqueName: \"kubernetes.io/projected/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-kube-api-access-5n2sd\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:44.993549 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.993495 5871 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0174771e-0c4e-4d4d-9f5f-a260a6719f1a-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:44.993549 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:44.993505 5871 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f310886-2c25-4b42-b71c-2bbb20b2a4eb-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:45.547453 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.547409 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" event={"ID":"6e3e8158-f5c0-4dc2-80b1-257e1ee30487","Type":"ContainerDied","Data":"e52840e6aa56530568d71f050e2938f5b5307b30662d96e3a95448bd95f28a1a"} Nov 02 22:09:45.547453 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.547443 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a977ml" Nov 02 22:09:45.548092 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.547499 5871 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e52840e6aa56530568d71f050e2938f5b5307b30662d96e3a95448bd95f28a1a" Nov 02 22:09:45.550108 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.550080 5871 generic.go:356] "Generic (PLEG): container finished" podID="ff95d187-fe3c-44a6-bd58-be88e0cf2aae" containerID="8905ab3dd592b9c97e0013fa57ae1a6a6d851c6eab6353ed57e67b466f64c776" exitCode=0 Nov 02 22:09:45.550225 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.550149 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" event={"ID":"ff95d187-fe3c-44a6-bd58-be88e0cf2aae","Type":"ContainerDied","Data":"8905ab3dd592b9c97e0013fa57ae1a6a6d851c6eab6353ed57e67b466f64c776"} Nov 02 22:09:45.552360 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.552338 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" Nov 02 22:09:45.552489 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.552338 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qhj84" event={"ID":"7b57f0db-4c96-4440-a057-3b51619d9873","Type":"ContainerDied","Data":"edc2c32a05d1820fe233f2296dd08f0f7b5b8966d84f61f5ff239e875e023dbd"} Nov 02 22:09:45.552489 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.552463 5871 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edc2c32a05d1820fe233f2296dd08f0f7b5b8966d84f61f5ff239e875e023dbd" Nov 02 22:09:45.554586 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.554543 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" Nov 02 22:09:45.554696 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.554540 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfh6rs" event={"ID":"0174771e-0c4e-4d4d-9f5f-a260a6719f1a","Type":"ContainerDied","Data":"5ecf3c25a9bf3f760e5b37d22df73decc433f36f002f7bc46f80636c0fc88391"} Nov 02 22:09:45.554696 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.554664 5871 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ecf3c25a9bf3f760e5b37d22df73decc433f36f002f7bc46f80636c0fc88391" Nov 02 22:09:45.560535 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.560510 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" event={"ID":"6f310886-2c25-4b42-b71c-2bbb20b2a4eb","Type":"ContainerDied","Data":"2b865b85898045f6f27035dbf779f0e7b7e3f300ea62fa7c7748a11ce7915024"} Nov 02 22:09:45.560666 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.560542 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b7lvs" Nov 02 22:09:45.560666 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.560550 5871 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b865b85898045f6f27035dbf779f0e7b7e3f300ea62fa7c7748a11ce7915024" Nov 02 22:09:45.563101 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.563075 5871 generic.go:356] "Generic (PLEG): container finished" podID="d76ffd5f-9ea1-4165-b735-371ebc0bd59a" containerID="82b4d2201af1f9404a277cf16b5ff2e3b02b8b3a2924281cef799c511f9babb0" exitCode=0 Nov 02 22:09:45.563223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.563120 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" event={"ID":"d76ffd5f-9ea1-4165-b735-371ebc0bd59a","Type":"ContainerDied","Data":"82b4d2201af1f9404a277cf16b5ff2e3b02b8b3a2924281cef799c511f9babb0"} Nov 02 22:09:45.689217 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.689185 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" Nov 02 22:09:45.700937 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.700910 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a9d920-ed37-4c77-89e6-9c5396dff5e2-util\" (UniqueName: \"kubernetes.io/empty-dir/90a9d920-ed37-4c77-89e6-9c5396dff5e2-util\") pod \"90a9d920-ed37-4c77-89e6-9c5396dff5e2\" (UID: \"90a9d920-ed37-4c77-89e6-9c5396dff5e2\") " Nov 02 22:09:45.701129 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.701002 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a9d920-ed37-4c77-89e6-9c5396dff5e2-kube-api-access-sf7kb\" (UniqueName: \"kubernetes.io/projected/90a9d920-ed37-4c77-89e6-9c5396dff5e2-kube-api-access-sf7kb\") pod \"90a9d920-ed37-4c77-89e6-9c5396dff5e2\" (UID: \"90a9d920-ed37-4c77-89e6-9c5396dff5e2\") " Nov 02 22:09:45.701129 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.701035 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a9d920-ed37-4c77-89e6-9c5396dff5e2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/90a9d920-ed37-4c77-89e6-9c5396dff5e2-bundle\") pod \"90a9d920-ed37-4c77-89e6-9c5396dff5e2\" (UID: \"90a9d920-ed37-4c77-89e6-9c5396dff5e2\") " Nov 02 22:09:45.701578 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.701550 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a9d920-ed37-4c77-89e6-9c5396dff5e2-bundle" pod "90a9d920-ed37-4c77-89e6-9c5396dff5e2" (UID: "90a9d920-ed37-4c77-89e6-9c5396dff5e2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:45.704279 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.704253 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a9d920-ed37-4c77-89e6-9c5396dff5e2-kube-api-access-sf7kb" pod "90a9d920-ed37-4c77-89e6-9c5396dff5e2" (UID: "90a9d920-ed37-4c77-89e6-9c5396dff5e2"). InnerVolumeSpecName "kube-api-access-sf7kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:45.707670 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.707637 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a9d920-ed37-4c77-89e6-9c5396dff5e2-util" pod "90a9d920-ed37-4c77-89e6-9c5396dff5e2" (UID: "90a9d920-ed37-4c77-89e6-9c5396dff5e2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:45.730981 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.730948 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" Nov 02 22:09:45.779149 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.779119 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" Nov 02 22:09:45.782046 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.782024 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" Nov 02 22:09:45.802120 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.802081 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-util\" (UniqueName: \"kubernetes.io/empty-dir/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-util\") pod \"d76ffd5f-9ea1-4165-b735-371ebc0bd59a\" (UID: \"d76ffd5f-9ea1-4165-b735-371ebc0bd59a\") " Nov 02 22:09:45.802120 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.802120 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01be6b4-9954-414a-a080-1a438c467ff3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f01be6b4-9954-414a-a080-1a438c467ff3-bundle\") pod \"f01be6b4-9954-414a-a080-1a438c467ff3\" (UID: \"f01be6b4-9954-414a-a080-1a438c467ff3\") " Nov 02 22:09:45.802375 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.802143 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4318c2-0167-44ea-bbe9-947f4a85783d-kube-api-access-gpskt\" (UniqueName: \"kubernetes.io/projected/fa4318c2-0167-44ea-bbe9-947f4a85783d-kube-api-access-gpskt\") pod \"fa4318c2-0167-44ea-bbe9-947f4a85783d\" (UID: \"fa4318c2-0167-44ea-bbe9-947f4a85783d\") " Nov 02 22:09:45.802375 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.802170 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01be6b4-9954-414a-a080-1a438c467ff3-util\" (UniqueName: \"kubernetes.io/empty-dir/f01be6b4-9954-414a-a080-1a438c467ff3-util\") pod \"f01be6b4-9954-414a-a080-1a438c467ff3\" (UID: \"f01be6b4-9954-414a-a080-1a438c467ff3\") " Nov 02 22:09:45.802375 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.802199 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01be6b4-9954-414a-a080-1a438c467ff3-kube-api-access-bflws\" (UniqueName: \"kubernetes.io/projected/f01be6b4-9954-414a-a080-1a438c467ff3-kube-api-access-bflws\") pod \"f01be6b4-9954-414a-a080-1a438c467ff3\" (UID: \"f01be6b4-9954-414a-a080-1a438c467ff3\") " Nov 02 22:09:45.802375 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.802254 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-kube-api-access-s9l68\" (UniqueName: \"kubernetes.io/projected/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-kube-api-access-s9l68\") pod \"d76ffd5f-9ea1-4165-b735-371ebc0bd59a\" (UID: \"d76ffd5f-9ea1-4165-b735-371ebc0bd59a\") " Nov 02 22:09:45.802375 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.802288 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-bundle\") pod \"d76ffd5f-9ea1-4165-b735-371ebc0bd59a\" (UID: \"d76ffd5f-9ea1-4165-b735-371ebc0bd59a\") " Nov 02 22:09:45.802375 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.802319 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4318c2-0167-44ea-bbe9-947f4a85783d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa4318c2-0167-44ea-bbe9-947f4a85783d-bundle\") pod \"fa4318c2-0167-44ea-bbe9-947f4a85783d\" (UID: \"fa4318c2-0167-44ea-bbe9-947f4a85783d\") " Nov 02 22:09:45.802375 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.802362 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa4318c2-0167-44ea-bbe9-947f4a85783d-util\" (UniqueName: \"kubernetes.io/empty-dir/fa4318c2-0167-44ea-bbe9-947f4a85783d-util\") pod \"fa4318c2-0167-44ea-bbe9-947f4a85783d\" (UID: \"fa4318c2-0167-44ea-bbe9-947f4a85783d\") " Nov 02 22:09:45.802720 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.802631 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf7kb\" (UniqueName: \"kubernetes.io/projected/90a9d920-ed37-4c77-89e6-9c5396dff5e2-kube-api-access-sf7kb\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:45.802720 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.802653 5871 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90a9d920-ed37-4c77-89e6-9c5396dff5e2-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:45.802720 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.802667 5871 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90a9d920-ed37-4c77-89e6-9c5396dff5e2-util\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:45.802994 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.802946 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01be6b4-9954-414a-a080-1a438c467ff3-bundle" pod "f01be6b4-9954-414a-a080-1a438c467ff3" (UID: "f01be6b4-9954-414a-a080-1a438c467ff3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:45.803277 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.803256 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4318c2-0167-44ea-bbe9-947f4a85783d-bundle" pod "fa4318c2-0167-44ea-bbe9-947f4a85783d" (UID: "fa4318c2-0167-44ea-bbe9-947f4a85783d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:45.803513 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.803483 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-bundle" pod "d76ffd5f-9ea1-4165-b735-371ebc0bd59a" (UID: "d76ffd5f-9ea1-4165-b735-371ebc0bd59a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:45.804770 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.804740 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4318c2-0167-44ea-bbe9-947f4a85783d-kube-api-access-gpskt" pod "fa4318c2-0167-44ea-bbe9-947f4a85783d" (UID: "fa4318c2-0167-44ea-bbe9-947f4a85783d"). InnerVolumeSpecName "kube-api-access-gpskt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:45.804910 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.804840 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01be6b4-9954-414a-a080-1a438c467ff3-kube-api-access-bflws" pod "f01be6b4-9954-414a-a080-1a438c467ff3" (UID: "f01be6b4-9954-414a-a080-1a438c467ff3"). InnerVolumeSpecName "kube-api-access-bflws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:45.805270 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.805244 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-kube-api-access-s9l68" pod "d76ffd5f-9ea1-4165-b735-371ebc0bd59a" (UID: "d76ffd5f-9ea1-4165-b735-371ebc0bd59a"). InnerVolumeSpecName "kube-api-access-s9l68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:45.808640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.808617 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01be6b4-9954-414a-a080-1a438c467ff3-util" pod "f01be6b4-9954-414a-a080-1a438c467ff3" (UID: "f01be6b4-9954-414a-a080-1a438c467ff3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:45.810238 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.810215 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-util" pod "d76ffd5f-9ea1-4165-b735-371ebc0bd59a" (UID: "d76ffd5f-9ea1-4165-b735-371ebc0bd59a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:45.810833 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.810809 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4318c2-0167-44ea-bbe9-947f4a85783d-util" pod "fa4318c2-0167-44ea-bbe9-947f4a85783d" (UID: "fa4318c2-0167-44ea-bbe9-947f4a85783d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:45.904090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.904001 5871 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f01be6b4-9954-414a-a080-1a438c467ff3-util\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:45.904090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.904032 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bflws\" (UniqueName: \"kubernetes.io/projected/f01be6b4-9954-414a-a080-1a438c467ff3-kube-api-access-bflws\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:45.904090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.904044 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9l68\" (UniqueName: \"kubernetes.io/projected/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-kube-api-access-s9l68\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:45.904090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.904053 5871 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:45.904090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.904062 5871 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa4318c2-0167-44ea-bbe9-947f4a85783d-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:45.904090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.904070 5871 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa4318c2-0167-44ea-bbe9-947f4a85783d-util\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:45.904090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.904078 5871 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d76ffd5f-9ea1-4165-b735-371ebc0bd59a-util\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:45.904090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.904085 5871 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f01be6b4-9954-414a-a080-1a438c467ff3-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:45.904090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:45.904094 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpskt\" (UniqueName: \"kubernetes.io/projected/fa4318c2-0167-44ea-bbe9-947f4a85783d-kube-api-access-gpskt\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:46.570001 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.569974 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" Nov 02 22:09:46.570481 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.569972 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kn6x8" event={"ID":"f01be6b4-9954-414a-a080-1a438c467ff3","Type":"ContainerDied","Data":"ca756e99e9c849a5d0bd0234ec13be80e6311fa2f12ce0be69f98eff511e963e"} Nov 02 22:09:46.570481 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.570098 5871 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca756e99e9c849a5d0bd0234ec13be80e6311fa2f12ce0be69f98eff511e963e" Nov 02 22:09:46.571993 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.571964 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" event={"ID":"d76ffd5f-9ea1-4165-b735-371ebc0bd59a","Type":"ContainerDied","Data":"1bc15d57bec61020fb730ac8eb9dd1644293c1a89cb3ad60e58d198117c8229e"} Nov 02 22:09:46.571993 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.571983 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw6plx" Nov 02 22:09:46.572172 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.572012 5871 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bc15d57bec61020fb730ac8eb9dd1644293c1a89cb3ad60e58d198117c8229e" Nov 02 22:09:46.573860 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.573839 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" Nov 02 22:09:46.574000 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.573862 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9lz4g" event={"ID":"90a9d920-ed37-4c77-89e6-9c5396dff5e2","Type":"ContainerDied","Data":"50769e929f532826575d982a733ba281907f3a6f685d2fe6428c4e01410577b7"} Nov 02 22:09:46.574000 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.573918 5871 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50769e929f532826575d982a733ba281907f3a6f685d2fe6428c4e01410577b7" Nov 02 22:09:46.575892 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.575848 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" Nov 02 22:09:46.575986 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.575907 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l2s8q" event={"ID":"fa4318c2-0167-44ea-bbe9-947f4a85783d","Type":"ContainerDied","Data":"41b079f6207d6f91265c9937c042924eb17e4b0d26a70c5a4ac3a05911a5c46f"} Nov 02 22:09:46.575986 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.575953 5871 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41b079f6207d6f91265c9937c042924eb17e4b0d26a70c5a4ac3a05911a5c46f" Nov 02 22:09:46.699532 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.699505 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" Nov 02 22:09:46.809860 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.809817 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-bundle\") pod \"ff95d187-fe3c-44a6-bd58-be88e0cf2aae\" (UID: \"ff95d187-fe3c-44a6-bd58-be88e0cf2aae\") " Nov 02 22:09:46.810183 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.809926 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-kube-api-access-tmpd6\" (UniqueName: \"kubernetes.io/projected/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-kube-api-access-tmpd6\") pod \"ff95d187-fe3c-44a6-bd58-be88e0cf2aae\" (UID: \"ff95d187-fe3c-44a6-bd58-be88e0cf2aae\") " Nov 02 22:09:46.810183 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.809984 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-util\" (UniqueName: \"kubernetes.io/empty-dir/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-util\") pod \"ff95d187-fe3c-44a6-bd58-be88e0cf2aae\" (UID: \"ff95d187-fe3c-44a6-bd58-be88e0cf2aae\") " Nov 02 22:09:46.810521 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.810493 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-bundle" pod "ff95d187-fe3c-44a6-bd58-be88e0cf2aae" (UID: "ff95d187-fe3c-44a6-bd58-be88e0cf2aae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:46.812105 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.812078 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-kube-api-access-tmpd6" pod "ff95d187-fe3c-44a6-bd58-be88e0cf2aae" (UID: "ff95d187-fe3c-44a6-bd58-be88e0cf2aae"). InnerVolumeSpecName "kube-api-access-tmpd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:46.815849 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.815828 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-util" pod "ff95d187-fe3c-44a6-bd58-be88e0cf2aae" (UID: "ff95d187-fe3c-44a6-bd58-be88e0cf2aae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:46.911255 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.911220 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmpd6\" (UniqueName: \"kubernetes.io/projected/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-kube-api-access-tmpd6\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:46.911255 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.911251 5871 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-util\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:46.911255 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:46.911262 5871 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff95d187-fe3c-44a6-bd58-be88e0cf2aae-bundle\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:47.582312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:47.582227 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" Nov 02 22:09:47.582312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:47.582247 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71sbq2w" event={"ID":"ff95d187-fe3c-44a6-bd58-be88e0cf2aae","Type":"ContainerDied","Data":"eff9d69b155f3d0f8fc5e5eecc804cbd598788f15d44272274fcffbb894e7e9a"} Nov 02 22:09:47.582312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:47.582297 5871 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eff9d69b155f3d0f8fc5e5eecc804cbd598788f15d44272274fcffbb894e7e9a" Nov 02 22:09:49.672410 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:49.672372 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:52.015556 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015519 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lpt86"] Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015808 5871 cpu_manager.go:414] "containerMap: removing container" podUID="fa4318c2-0167-44ea-bbe9-947f4a85783d" containerName="util" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015819 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4318c2-0167-44ea-bbe9-947f4a85783d" containerName="util" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015832 5871 cpu_manager.go:414] "containerMap: removing container" podUID="6f310886-2c25-4b42-b71c-2bbb20b2a4eb" containerName="util" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015837 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f310886-2c25-4b42-b71c-2bbb20b2a4eb" containerName="util" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015845 5871 cpu_manager.go:414] "containerMap: removing container" podUID="ff95d187-fe3c-44a6-bd58-be88e0cf2aae" containerName="extract" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015850 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff95d187-fe3c-44a6-bd58-be88e0cf2aae" containerName="extract" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015859 5871 cpu_manager.go:414] "containerMap: removing container" podUID="0174771e-0c4e-4d4d-9f5f-a260a6719f1a" containerName="extract" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015864 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0174771e-0c4e-4d4d-9f5f-a260a6719f1a" containerName="extract" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015885 5871 cpu_manager.go:414] "containerMap: removing container" podUID="6e3e8158-f5c0-4dc2-80b1-257e1ee30487" containerName="extract" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015891 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3e8158-f5c0-4dc2-80b1-257e1ee30487" containerName="extract" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015897 5871 cpu_manager.go:414] "containerMap: removing container" podUID="7b57f0db-4c96-4440-a057-3b51619d9873" containerName="pull" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015902 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b57f0db-4c96-4440-a057-3b51619d9873" containerName="pull" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015908 5871 cpu_manager.go:414] "containerMap: removing container" podUID="fa4318c2-0167-44ea-bbe9-947f4a85783d" containerName="extract" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015912 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4318c2-0167-44ea-bbe9-947f4a85783d" containerName="extract" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015918 5871 cpu_manager.go:414] "containerMap: removing container" podUID="d76ffd5f-9ea1-4165-b735-371ebc0bd59a" containerName="extract" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015923 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76ffd5f-9ea1-4165-b735-371ebc0bd59a" containerName="extract" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015932 5871 cpu_manager.go:414] "containerMap: removing container" podUID="90a9d920-ed37-4c77-89e6-9c5396dff5e2" containerName="extract" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015937 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a9d920-ed37-4c77-89e6-9c5396dff5e2" containerName="extract" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015947 5871 cpu_manager.go:414] "containerMap: removing container" podUID="0174771e-0c4e-4d4d-9f5f-a260a6719f1a" containerName="util" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015951 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0174771e-0c4e-4d4d-9f5f-a260a6719f1a" containerName="util" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015956 5871 cpu_manager.go:414] "containerMap: removing container" podUID="d5009cc4-ddbc-4952-a2c6-6da12f2428e2" containerName="util" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015961 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5009cc4-ddbc-4952-a2c6-6da12f2428e2" containerName="util" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015968 5871 cpu_manager.go:414] "containerMap: removing container" podUID="90a9d920-ed37-4c77-89e6-9c5396dff5e2" containerName="util" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015973 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a9d920-ed37-4c77-89e6-9c5396dff5e2" containerName="util" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015978 5871 cpu_manager.go:414] "containerMap: removing container" podUID="58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df" containerName="pull" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015982 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df" containerName="pull" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015988 5871 cpu_manager.go:414] "containerMap: removing container" podUID="f01be6b4-9954-414a-a080-1a438c467ff3" containerName="pull" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015993 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01be6b4-9954-414a-a080-1a438c467ff3" containerName="pull" Nov 02 22:09:52.015984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.015999 5871 cpu_manager.go:414] "containerMap: removing container" podUID="6e3e8158-f5c0-4dc2-80b1-257e1ee30487" containerName="pull" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016003 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3e8158-f5c0-4dc2-80b1-257e1ee30487" containerName="pull" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016011 5871 cpu_manager.go:414] "containerMap: removing container" podUID="7b57f0db-4c96-4440-a057-3b51619d9873" containerName="util" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016015 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b57f0db-4c96-4440-a057-3b51619d9873" containerName="util" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016022 5871 cpu_manager.go:414] "containerMap: removing container" podUID="58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016027 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016035 5871 cpu_manager.go:414] "containerMap: removing container" podUID="fa4318c2-0167-44ea-bbe9-947f4a85783d" containerName="pull" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016039 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4318c2-0167-44ea-bbe9-947f4a85783d" containerName="pull" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016045 5871 cpu_manager.go:414] "containerMap: removing container" podUID="f01be6b4-9954-414a-a080-1a438c467ff3" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016049 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01be6b4-9954-414a-a080-1a438c467ff3" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016056 5871 cpu_manager.go:414] "containerMap: removing container" podUID="58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df" containerName="util" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016060 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df" containerName="util" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016067 5871 cpu_manager.go:414] "containerMap: removing container" podUID="f01be6b4-9954-414a-a080-1a438c467ff3" containerName="util" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016071 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01be6b4-9954-414a-a080-1a438c467ff3" containerName="util" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016076 5871 cpu_manager.go:414] "containerMap: removing container" podUID="90a9d920-ed37-4c77-89e6-9c5396dff5e2" containerName="pull" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016080 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a9d920-ed37-4c77-89e6-9c5396dff5e2" containerName="pull" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016085 5871 cpu_manager.go:414] "containerMap: removing container" podUID="ff95d187-fe3c-44a6-bd58-be88e0cf2aae" containerName="util" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016089 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff95d187-fe3c-44a6-bd58-be88e0cf2aae" containerName="util" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016095 5871 cpu_manager.go:414] "containerMap: removing container" podUID="7b57f0db-4c96-4440-a057-3b51619d9873" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016099 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b57f0db-4c96-4440-a057-3b51619d9873" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016105 5871 cpu_manager.go:414] "containerMap: removing container" podUID="d5009cc4-ddbc-4952-a2c6-6da12f2428e2" containerName="pull" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016109 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5009cc4-ddbc-4952-a2c6-6da12f2428e2" containerName="pull" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016115 5871 cpu_manager.go:414] "containerMap: removing container" podUID="0174771e-0c4e-4d4d-9f5f-a260a6719f1a" containerName="pull" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016119 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0174771e-0c4e-4d4d-9f5f-a260a6719f1a" containerName="pull" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016126 5871 cpu_manager.go:414] "containerMap: removing container" podUID="d76ffd5f-9ea1-4165-b735-371ebc0bd59a" containerName="util" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016130 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76ffd5f-9ea1-4165-b735-371ebc0bd59a" containerName="util" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016136 5871 cpu_manager.go:414] "containerMap: removing container" podUID="6f310886-2c25-4b42-b71c-2bbb20b2a4eb" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016140 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f310886-2c25-4b42-b71c-2bbb20b2a4eb" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016147 5871 cpu_manager.go:414] "containerMap: removing container" podUID="6f310886-2c25-4b42-b71c-2bbb20b2a4eb" containerName="pull" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016152 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f310886-2c25-4b42-b71c-2bbb20b2a4eb" containerName="pull" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016159 5871 cpu_manager.go:414] "containerMap: removing container" podUID="d76ffd5f-9ea1-4165-b735-371ebc0bd59a" containerName="pull" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016163 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76ffd5f-9ea1-4165-b735-371ebc0bd59a" containerName="pull" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016170 5871 cpu_manager.go:414] "containerMap: removing container" podUID="6e3e8158-f5c0-4dc2-80b1-257e1ee30487" containerName="util" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016174 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3e8158-f5c0-4dc2-80b1-257e1ee30487" containerName="util" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016180 5871 cpu_manager.go:414] "containerMap: removing container" podUID="ff95d187-fe3c-44a6-bd58-be88e0cf2aae" containerName="pull" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016184 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff95d187-fe3c-44a6-bd58-be88e0cf2aae" containerName="pull" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016191 5871 cpu_manager.go:414] "containerMap: removing container" podUID="d5009cc4-ddbc-4952-a2c6-6da12f2428e2" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016196 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5009cc4-ddbc-4952-a2c6-6da12f2428e2" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016259 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f310886-2c25-4b42-b71c-2bbb20b2a4eb" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016268 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5009cc4-ddbc-4952-a2c6-6da12f2428e2" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016273 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff95d187-fe3c-44a6-bd58-be88e0cf2aae" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016279 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="0174771e-0c4e-4d4d-9f5f-a260a6719f1a" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016284 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3e8158-f5c0-4dc2-80b1-257e1ee30487" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016289 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4318c2-0167-44ea-bbe9-947f4a85783d" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016294 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="d76ffd5f-9ea1-4165-b735-371ebc0bd59a" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016300 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a9d920-ed37-4c77-89e6-9c5396dff5e2" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016304 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01be6b4-9954-414a-a080-1a438c467ff3" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016310 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="58fd6d4e-bdb2-4d51-89ff-7b0fef90f0df" containerName="extract" Nov 02 22:09:52.016794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.016315 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b57f0db-4c96-4440-a057-3b51619d9873" containerName="extract" Nov 02 22:09:52.018452 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.018434 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lpt86" Nov 02 22:09:52.031579 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.031543 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:52.055800 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.055770 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fr7z\" (UniqueName: \"kubernetes.io/projected/9de728ab-24e5-46b7-9fd2-9bf0f7887251-kube-api-access-6fr7z\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lpt86\" (UID: \"9de728ab-24e5-46b7-9fd2-9bf0f7887251\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lpt86" Nov 02 22:09:52.055966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.055940 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9de728ab-24e5-46b7-9fd2-9bf0f7887251-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lpt86\" (UID: \"9de728ab-24e5-46b7-9fd2-9bf0f7887251\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lpt86" Nov 02 22:09:52.115404 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.115369 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 22:09:52.115404 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.115363 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:09:52.115867 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.115442 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-mwt8k\"" Nov 02 22:09:52.116791 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.116768 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lpt86"] Nov 02 22:09:52.162219 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.157695 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9de728ab-24e5-46b7-9fd2-9bf0f7887251-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lpt86\" (UID: \"9de728ab-24e5-46b7-9fd2-9bf0f7887251\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lpt86" Nov 02 22:09:52.162219 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.157850 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fr7z\" (UniqueName: \"kubernetes.io/projected/9de728ab-24e5-46b7-9fd2-9bf0f7887251-kube-api-access-6fr7z\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lpt86\" (UID: \"9de728ab-24e5-46b7-9fd2-9bf0f7887251\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lpt86" Nov 02 22:09:52.162219 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.158109 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9de728ab-24e5-46b7-9fd2-9bf0f7887251-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lpt86\" (UID: \"9de728ab-24e5-46b7-9fd2-9bf0f7887251\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lpt86" Nov 02 22:09:52.215290 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.215260 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fr7z\" (UniqueName: \"kubernetes.io/projected/9de728ab-24e5-46b7-9fd2-9bf0f7887251-kube-api-access-6fr7z\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lpt86\" (UID: \"9de728ab-24e5-46b7-9fd2-9bf0f7887251\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lpt86" Nov 02 22:09:52.329619 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.329526 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lpt86" Nov 02 22:09:52.753883 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:52.753846 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lpt86"] Nov 02 22:09:52.761487 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:09:52.761457 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de728ab_24e5_46b7_9fd2_9bf0f7887251.slice/crio-44e8220ca35978f8e3021e33a53e7bd0e1c89e94029950805ecacbeb0bf34d5f WatchSource:0}: Error finding container 44e8220ca35978f8e3021e33a53e7bd0e1c89e94029950805ecacbeb0bf34d5f: Status 404 returned error can't find the container with id 44e8220ca35978f8e3021e33a53e7bd0e1c89e94029950805ecacbeb0bf34d5f Nov 02 22:09:53.608754 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:53.608717 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lpt86" event={"ID":"9de728ab-24e5-46b7-9fd2-9bf0f7887251","Type":"ContainerStarted","Data":"44e8220ca35978f8e3021e33a53e7bd0e1c89e94029950805ecacbeb0bf34d5f"} Nov 02 22:09:53.688898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:53.688847 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:53.861738 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:53.861660 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:56.270315 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:56.270224 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:56.625563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:56.625532 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lpt86" event={"ID":"9de728ab-24e5-46b7-9fd2-9bf0f7887251","Type":"ContainerStarted","Data":"2c9a878329ea9aa6bd9b4449a4c1bcc5f22146403b51419ab34fc0f3750ebd69"} Nov 02 22:09:56.831185 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:56.831129 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lpt86" podStartSLOduration=2.62798782 podStartE2EDuration="5.831114924s" totalImagesPullingTime="3.203127104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:56.783866368 +0000 UTC m=+219.596196547" watchObservedRunningTime="2026-11-02 22:09:56.831114924 +0000 UTC m=+219.643445106" Nov 02 22:09:57.360299 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:57.360262 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:57.958912 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:57.958854 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:58.156309 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:58.156268 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:58.351072 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:58.350989 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:59.579236 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:59.579199 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wn5jz"] Nov 02 22:09:59.582560 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:59.582531 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wn5jz" Nov 02 22:09:59.620039 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:59.619998 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a0ad59b-8818-47f2-bdd2-beb37d6d8b6f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wn5jz\" (UID: \"2a0ad59b-8818-47f2-bdd2-beb37d6d8b6f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wn5jz" Nov 02 22:09:59.620218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:59.620154 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwrwn\" (UniqueName: \"kubernetes.io/projected/2a0ad59b-8818-47f2-bdd2-beb37d6d8b6f-kube-api-access-vwrwn\") pod \"cert-manager-cainjector-5c57bbd4fc-wn5jz\" (UID: \"2a0ad59b-8818-47f2-bdd2-beb37d6d8b6f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wn5jz" Nov 02 22:09:59.631449 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:59.631394 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 22:09:59.677320 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:59.677288 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 22:09:59.677501 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:59.677368 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-pd5n5\"" Nov 02 22:09:59.678974 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:59.678951 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wn5jz"] Nov 02 22:09:59.717277 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:59.717243 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:09:59.721451 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:59.721429 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwrwn\" (UniqueName: \"kubernetes.io/projected/2a0ad59b-8818-47f2-bdd2-beb37d6d8b6f-kube-api-access-vwrwn\") pod \"cert-manager-cainjector-5c57bbd4fc-wn5jz\" (UID: \"2a0ad59b-8818-47f2-bdd2-beb37d6d8b6f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wn5jz" Nov 02 22:09:59.721530 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:59.721470 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a0ad59b-8818-47f2-bdd2-beb37d6d8b6f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wn5jz\" (UID: \"2a0ad59b-8818-47f2-bdd2-beb37d6d8b6f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wn5jz" Nov 02 22:09:59.778893 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:59.778845 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwrwn\" (UniqueName: \"kubernetes.io/projected/2a0ad59b-8818-47f2-bdd2-beb37d6d8b6f-kube-api-access-vwrwn\") pod \"cert-manager-cainjector-5c57bbd4fc-wn5jz\" (UID: \"2a0ad59b-8818-47f2-bdd2-beb37d6d8b6f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wn5jz" Nov 02 22:09:59.779041 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:59.778891 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a0ad59b-8818-47f2-bdd2-beb37d6d8b6f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wn5jz\" (UID: \"2a0ad59b-8818-47f2-bdd2-beb37d6d8b6f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wn5jz" Nov 02 22:09:59.904346 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:09:59.904310 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wn5jz" Nov 02 22:10:00.274776 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:00.274698 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wn5jz"] Nov 02 22:10:00.282488 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:10:00.282457 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a0ad59b_8818_47f2_bdd2_beb37d6d8b6f.slice/crio-21a9887273135746ccafd08d045f8114a3c60245ec5caa6a0b5f3949cbd847b0 WatchSource:0}: Error finding container 21a9887273135746ccafd08d045f8114a3c60245ec5caa6a0b5f3949cbd847b0: Status 404 returned error can't find the container with id 21a9887273135746ccafd08d045f8114a3c60245ec5caa6a0b5f3949cbd847b0 Nov 02 22:10:00.651039 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:00.650999 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wn5jz" event={"ID":"2a0ad59b-8818-47f2-bdd2-beb37d6d8b6f","Type":"ContainerStarted","Data":"21a9887273135746ccafd08d045f8114a3c60245ec5caa6a0b5f3949cbd847b0"} Nov 02 22:10:00.812958 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:00.812927 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:00.995897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:00.995789 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:01.693338 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:01.693305 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-p9229"] Nov 02 22:10:01.698078 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:01.698050 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p9229" Nov 02 22:10:01.741135 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:01.741091 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bss6m\" (UniqueName: \"kubernetes.io/projected/aff64e43-f4d1-4e2e-927a-c83ad7861193-kube-api-access-bss6m\") pod \"authorino-operator-7b7bb4bc77-p9229\" (UID: \"aff64e43-f4d1-4e2e-927a-c83ad7861193\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-p9229" Nov 02 22:10:01.746793 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:01.746761 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 22:10:01.792707 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:01.792681 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-rnq7r\"" Nov 02 22:10:01.792707 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:01.792685 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 22:10:01.794173 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:01.794153 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-p9229"] Nov 02 22:10:01.842090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:01.842049 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bss6m\" (UniqueName: \"kubernetes.io/projected/aff64e43-f4d1-4e2e-927a-c83ad7861193-kube-api-access-bss6m\") pod \"authorino-operator-7b7bb4bc77-p9229\" (UID: \"aff64e43-f4d1-4e2e-927a-c83ad7861193\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-p9229" Nov 02 22:10:01.902999 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:01.902971 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bss6m\" (UniqueName: \"kubernetes.io/projected/aff64e43-f4d1-4e2e-927a-c83ad7861193-kube-api-access-bss6m\") pod \"authorino-operator-7b7bb4bc77-p9229\" (UID: \"aff64e43-f4d1-4e2e-927a-c83ad7861193\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-p9229" Nov 02 22:10:02.011030 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:02.010940 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p9229" Nov 02 22:10:02.342365 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:02.342272 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:02.389630 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:02.389595 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-p9229"] Nov 02 22:10:02.397092 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:10:02.397049 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff64e43_f4d1_4e2e_927a_c83ad7861193.slice/crio-4d9869cb909215d9f2c90c11c52b3bc12e9eefbd14eab7233a2b7e6cc1205250 WatchSource:0}: Error finding container 4d9869cb909215d9f2c90c11c52b3bc12e9eefbd14eab7233a2b7e6cc1205250: Status 404 returned error can't find the container with id 4d9869cb909215d9f2c90c11c52b3bc12e9eefbd14eab7233a2b7e6cc1205250 Nov 02 22:10:02.520171 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:02.520133 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:02.663759 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:02.663719 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p9229" event={"ID":"aff64e43-f4d1-4e2e-927a-c83ad7861193","Type":"ContainerStarted","Data":"4d9869cb909215d9f2c90c11c52b3bc12e9eefbd14eab7233a2b7e6cc1205250"} Nov 02 22:10:03.471096 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:03.471051 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:04.676303 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:04.676263 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wn5jz" event={"ID":"2a0ad59b-8818-47f2-bdd2-beb37d6d8b6f","Type":"ContainerStarted","Data":"72a7a1fe5bb0703ffd0dda67e2b2d30b830a727bbd4d49d6d6b670b9854e3ea0"} Nov 02 22:10:05.152150 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.152090 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wn5jz" podStartSLOduration=2.234707038 podStartE2EDuration="6.152065446s" totalImagesPullingTime="3.917358408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:04.784378886 +0000 UTC m=+227.596709065" watchObservedRunningTime="2026-11-02 22:10:05.152065446 +0000 UTC m=+227.964395621" Nov 02 22:10:05.152362 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.152264 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9g6g9"] Nov 02 22:10:05.155680 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.155651 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g6g9" Nov 02 22:10:05.204483 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.204449 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:10:05.204653 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.204566 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-hdzz4\"" Nov 02 22:10:05.250552 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.250515 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 22:10:05.252399 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.252364 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9g6g9"] Nov 02 22:10:05.274264 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.274220 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19e1a752-5e8a-45a8-bc96-922e98d1cab0-tmp\") pod \"jobset-operator-674cd59dc5-9g6g9\" (UID: \"19e1a752-5e8a-45a8-bc96-922e98d1cab0\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g6g9" Nov 02 22:10:05.274442 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.274370 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd69j\" (UniqueName: \"kubernetes.io/projected/19e1a752-5e8a-45a8-bc96-922e98d1cab0-kube-api-access-fd69j\") pod \"jobset-operator-674cd59dc5-9g6g9\" (UID: \"19e1a752-5e8a-45a8-bc96-922e98d1cab0\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g6g9" Nov 02 22:10:05.375737 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.375701 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19e1a752-5e8a-45a8-bc96-922e98d1cab0-tmp\") pod \"jobset-operator-674cd59dc5-9g6g9\" (UID: \"19e1a752-5e8a-45a8-bc96-922e98d1cab0\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g6g9" Nov 02 22:10:05.376011 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.375780 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd69j\" (UniqueName: \"kubernetes.io/projected/19e1a752-5e8a-45a8-bc96-922e98d1cab0-kube-api-access-fd69j\") pod \"jobset-operator-674cd59dc5-9g6g9\" (UID: \"19e1a752-5e8a-45a8-bc96-922e98d1cab0\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g6g9" Nov 02 22:10:05.376110 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.376087 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19e1a752-5e8a-45a8-bc96-922e98d1cab0-tmp\") pod \"jobset-operator-674cd59dc5-9g6g9\" (UID: \"19e1a752-5e8a-45a8-bc96-922e98d1cab0\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g6g9" Nov 02 22:10:05.434704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.434630 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd69j\" (UniqueName: \"kubernetes.io/projected/19e1a752-5e8a-45a8-bc96-922e98d1cab0-kube-api-access-fd69j\") pod \"jobset-operator-674cd59dc5-9g6g9\" (UID: \"19e1a752-5e8a-45a8-bc96-922e98d1cab0\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g6g9" Nov 02 22:10:05.435293 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.435264 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr"] Nov 02 22:10:05.445366 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.445336 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" Nov 02 22:10:05.470048 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.470012 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g6g9" Nov 02 22:10:05.494034 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.493998 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr"] Nov 02 22:10:05.498132 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.498099 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gjbrx\"" Nov 02 22:10:05.577490 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.577448 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbq2h\" (UniqueName: \"kubernetes.io/projected/6b7ba2e0-b642-4b20-896b-e79253436f13-kube-api-access-xbq2h\") pod \"cert-manager-webhook-5dbbbf75cc-z4rhr\" (UID: \"6b7ba2e0-b642-4b20-896b-e79253436f13\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" Nov 02 22:10:05.577698 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.577612 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b7ba2e0-b642-4b20-896b-e79253436f13-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-z4rhr\" (UID: \"6b7ba2e0-b642-4b20-896b-e79253436f13\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" Nov 02 22:10:05.678205 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.678159 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b7ba2e0-b642-4b20-896b-e79253436f13-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-z4rhr\" (UID: \"6b7ba2e0-b642-4b20-896b-e79253436f13\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" Nov 02 22:10:05.678715 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.678239 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbq2h\" (UniqueName: \"kubernetes.io/projected/6b7ba2e0-b642-4b20-896b-e79253436f13-kube-api-access-xbq2h\") pod \"cert-manager-webhook-5dbbbf75cc-z4rhr\" (UID: \"6b7ba2e0-b642-4b20-896b-e79253436f13\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" Nov 02 22:10:05.736007 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.735924 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b7ba2e0-b642-4b20-896b-e79253436f13-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-z4rhr\" (UID: \"6b7ba2e0-b642-4b20-896b-e79253436f13\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" Nov 02 22:10:05.736165 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.736041 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbq2h\" (UniqueName: \"kubernetes.io/projected/6b7ba2e0-b642-4b20-896b-e79253436f13-kube-api-access-xbq2h\") pod \"cert-manager-webhook-5dbbbf75cc-z4rhr\" (UID: \"6b7ba2e0-b642-4b20-896b-e79253436f13\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" Nov 02 22:10:05.760397 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.760353 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" Nov 02 22:10:05.854107 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:05.854071 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9g6g9"] Nov 02 22:10:05.861679 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:10:05.860787 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19e1a752_5e8a_45a8_bc96_922e98d1cab0.slice/crio-e6a87b904325031a52559e0e24023be3548dafea3e48f2b92874af1431718572 WatchSource:0}: Error finding container e6a87b904325031a52559e0e24023be3548dafea3e48f2b92874af1431718572: Status 404 returned error can't find the container with id e6a87b904325031a52559e0e24023be3548dafea3e48f2b92874af1431718572 Nov 02 22:10:06.147745 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:06.147702 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr"] Nov 02 22:10:06.155952 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:10:06.155921 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b7ba2e0_b642_4b20_896b_e79253436f13.slice/crio-68e48d90093f9c142b6176f182edb04828a730f483d3e6b16e86d17e0b9af956 WatchSource:0}: Error finding container 68e48d90093f9c142b6176f182edb04828a730f483d3e6b16e86d17e0b9af956: Status 404 returned error can't find the container with id 68e48d90093f9c142b6176f182edb04828a730f483d3e6b16e86d17e0b9af956 Nov 02 22:10:06.691389 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:06.691294 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" event={"ID":"6b7ba2e0-b642-4b20-896b-e79253436f13","Type":"ContainerStarted","Data":"cc1dbb5efc407f6d78fb8e385069630711ce31554e0ba4c8048168b3ad515dd9"} Nov 02 22:10:06.691389 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:06.691340 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" event={"ID":"6b7ba2e0-b642-4b20-896b-e79253436f13","Type":"ContainerStarted","Data":"68e48d90093f9c142b6176f182edb04828a730f483d3e6b16e86d17e0b9af956"} Nov 02 22:10:06.691941 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:06.691404 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" Nov 02 22:10:06.693095 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:06.693067 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g6g9" event={"ID":"19e1a752-5e8a-45a8-bc96-922e98d1cab0","Type":"ContainerStarted","Data":"e6a87b904325031a52559e0e24023be3548dafea3e48f2b92874af1431718572"} Nov 02 22:10:06.695193 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:06.695159 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p9229" event={"ID":"aff64e43-f4d1-4e2e-927a-c83ad7861193","Type":"ContainerStarted","Data":"4f853c6129c09868e91279a28f9db330cc4033f62b93bc790798ae802c56abe8"} Nov 02 22:10:06.695291 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:06.695264 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p9229" Nov 02 22:10:06.738693 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:06.738642 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:06.801960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:06.801907 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" podStartSLOduration=1.8018905570000001 podStartE2EDuration="1.801890557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:06.800257056 +0000 UTC m=+229.612587233" watchObservedRunningTime="2026-11-02 22:10:06.801890557 +0000 UTC m=+229.614220732" Nov 02 22:10:06.921600 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:06.921539 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p9229" podStartSLOduration=1.896727474 podStartE2EDuration="5.921523174s" totalImagesPullingTime="4.0247957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:06.920700542 +0000 UTC m=+229.733030756" watchObservedRunningTime="2026-11-02 22:10:06.921523174 +0000 UTC m=+229.733853342" Nov 02 22:10:06.934819 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:06.934779 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:07.297836 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:07.297805 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:07.540109 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:07.540056 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:07.735555 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:07.735518 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:07.941812 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:07.941776 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:08.130374 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:08.130335 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:09.112701 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.112632 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:09.380136 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.380027 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:09.592077 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.592028 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:09.617597 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.617548 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq"] Nov 02 22:10:09.624726 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.624693 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.712486 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.712454 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g6g9" event={"ID":"19e1a752-5e8a-45a8-bc96-922e98d1cab0","Type":"ContainerStarted","Data":"5946b49d7eecd978b90d499238d37f0d7feed9cdca2364f126796df5732ef38b"} Nov 02 22:10:09.715031 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.715005 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c51a9c56-798c-410c-b5f6-7b3379255a20-tmp\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.715146 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.715067 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5qqj\" (UniqueName: \"kubernetes.io/projected/c51a9c56-798c-410c-b5f6-7b3379255a20-kube-api-access-x5qqj\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.715201 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.715180 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c51a9c56-798c-410c-b5f6-7b3379255a20-webhook-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.715303 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.715213 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c51a9c56-798c-410c-b5f6-7b3379255a20-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.715303 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.715240 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c51a9c56-798c-410c-b5f6-7b3379255a20-manifests\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.715373 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.715350 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c51a9c56-798c-410c-b5f6-7b3379255a20-metrics-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.738018 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.737982 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 22:10:09.738018 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.738013 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 22:10:09.738228 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.738057 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 22:10:09.738228 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.738104 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-kldsf\"" Nov 02 22:10:09.738696 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.738670 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 22:10:09.738951 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.738937 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 22:10:09.762905 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.762854 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq"] Nov 02 22:10:09.800184 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.800132 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:09.816632 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.816600 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c51a9c56-798c-410c-b5f6-7b3379255a20-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.816924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.816665 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c51a9c56-798c-410c-b5f6-7b3379255a20-manifests\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.816924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.816702 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c51a9c56-798c-410c-b5f6-7b3379255a20-metrics-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.816924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.816768 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c51a9c56-798c-410c-b5f6-7b3379255a20-tmp\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.816924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.816836 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5qqj\" (UniqueName: \"kubernetes.io/projected/c51a9c56-798c-410c-b5f6-7b3379255a20-kube-api-access-x5qqj\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.816924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.816901 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c51a9c56-798c-410c-b5f6-7b3379255a20-webhook-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.818398 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.817408 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c51a9c56-798c-410c-b5f6-7b3379255a20-manifests\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.820273 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.820238 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c51a9c56-798c-410c-b5f6-7b3379255a20-metrics-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.820389 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.820238 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c51a9c56-798c-410c-b5f6-7b3379255a20-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.820461 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.820434 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c51a9c56-798c-410c-b5f6-7b3379255a20-webhook-cert\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.820516 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.820465 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c51a9c56-798c-410c-b5f6-7b3379255a20-tmp\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.877046 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.877011 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5qqj\" (UniqueName: \"kubernetes.io/projected/c51a9c56-798c-410c-b5f6-7b3379255a20-kube-api-access-x5qqj\") pod \"opendatahub-operator-controller-manager-d99fd484b-bqftq\" (UID: \"c51a9c56-798c-410c-b5f6-7b3379255a20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:09.910607 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.910502 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9g6g9" podStartSLOduration=1.52792439 podStartE2EDuration="4.910486148s" totalImagesPullingTime="3.382561758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:09.908718547 +0000 UTC m=+232.721048725" watchObservedRunningTime="2026-11-02 22:10:09.910486148 +0000 UTC m=+232.722816326" Nov 02 22:10:09.944151 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:09.944103 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:10.001961 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:10.001433 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:10.197934 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:10.197835 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:10.471504 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:10.471401 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq"] Nov 02 22:10:10.478015 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:10:10.477985 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51a9c56_798c_410c_b5f6_7b3379255a20.slice/crio-d8cdeed27df29b6768b3f15d9afd1f8aa474eba4f0559133b8c60066b4dc7653 WatchSource:0}: Error finding container d8cdeed27df29b6768b3f15d9afd1f8aa474eba4f0559133b8c60066b4dc7653: Status 404 returned error can't find the container with id d8cdeed27df29b6768b3f15d9afd1f8aa474eba4f0559133b8c60066b4dc7653 Nov 02 22:10:10.486672 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:10.486648 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:10.676171 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:10.676134 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:10.718604 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:10.718565 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" event={"ID":"c51a9c56-798c-410c-b5f6-7b3379255a20","Type":"ContainerStarted","Data":"d8cdeed27df29b6768b3f15d9afd1f8aa474eba4f0559133b8c60066b4dc7653"} Nov 02 22:10:12.702845 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:12.702809 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" Nov 02 22:10:13.544335 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:13.544295 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:13.731511 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:13.731471 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:13.772166 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:10:13.772101 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:10:14.741655 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:14.741614 5871 generic.go:356] "Generic (PLEG): container finished" podID="c51a9c56-798c-410c-b5f6-7b3379255a20" containerID="b608fd04be6eb313a638000cf0fbe33a2ab81dbf856b0f122d0d83823484bf9d" exitCode=0 Nov 02 22:10:14.742140 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:14.741688 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" event={"ID":"c51a9c56-798c-410c-b5f6-7b3379255a20","Type":"ContainerDied","Data":"b608fd04be6eb313a638000cf0fbe33a2ab81dbf856b0f122d0d83823484bf9d"} Nov 02 22:10:15.711487 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:15.711447 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:16.231614 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:16.231579 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:16.469015 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:16.468980 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:16.753671 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:16.753631 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" event={"ID":"c51a9c56-798c-410c-b5f6-7b3379255a20","Type":"ContainerStarted","Data":"75cdb7d4273ffb247aeb4d21e7829bb3b48986fbda88f75adc133c2baa9cd5ab"} Nov 02 22:10:16.753850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:16.753809 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:17.028134 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:17.028015 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" podStartSLOduration=2.931539706 podStartE2EDuration="8.027993382s" totalImagesPullingTime="4.6424863s" totalInitContainerRuntime="453.967376ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:16.981379387 +0000 UTC m=+239.793709566" watchObservedRunningTime="2026-11-02 22:10:17.027993382 +0000 UTC m=+239.840323560" Nov 02 22:10:17.701686 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:17.701653 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-p9229" Nov 02 22:10:17.931563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:17.931528 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-scqp5"] Nov 02 22:10:17.936416 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:17.936392 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-scqp5" Nov 02 22:10:17.988759 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:17.988661 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wvw9h\"" Nov 02 22:10:17.990601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:17.990572 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-scqp5"] Nov 02 22:10:18.093733 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:18.093687 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhmv9\" (UniqueName: \"kubernetes.io/projected/fb6fface-d61e-4b8b-8a0a-783123e5c1fe-kube-api-access-fhmv9\") pod \"limitador-operator-controller-manager-5778bb4449-scqp5\" (UID: \"fb6fface-d61e-4b8b-8a0a-783123e5c1fe\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-scqp5" Nov 02 22:10:18.195072 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:18.195028 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhmv9\" (UniqueName: \"kubernetes.io/projected/fb6fface-d61e-4b8b-8a0a-783123e5c1fe-kube-api-access-fhmv9\") pod \"limitador-operator-controller-manager-5778bb4449-scqp5\" (UID: \"fb6fface-d61e-4b8b-8a0a-783123e5c1fe\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-scqp5" Nov 02 22:10:18.253138 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:18.253061 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhmv9\" (UniqueName: \"kubernetes.io/projected/fb6fface-d61e-4b8b-8a0a-783123e5c1fe-kube-api-access-fhmv9\") pod \"limitador-operator-controller-manager-5778bb4449-scqp5\" (UID: \"fb6fface-d61e-4b8b-8a0a-783123e5c1fe\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-scqp5" Nov 02 22:10:18.549257 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:18.549157 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-scqp5" Nov 02 22:10:18.922961 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:18.922920 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-scqp5"] Nov 02 22:10:18.929933 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:10:18.929905 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6fface_d61e_4b8b_8a0a_783123e5c1fe.slice/crio-de4ac459bc81cb26d5a81ad892aad2de631c698d195ab4b1bbb647c8fb2c58d4 WatchSource:0}: Error finding container de4ac459bc81cb26d5a81ad892aad2de631c698d195ab4b1bbb647c8fb2c58d4: Status 404 returned error can't find the container with id de4ac459bc81cb26d5a81ad892aad2de631c698d195ab4b1bbb647c8fb2c58d4 Nov 02 22:10:19.090941 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:19.090896 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:19.268039 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:19.267955 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:19.697707 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:19.697677 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:19.768674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:19.768639 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-scqp5" event={"ID":"fb6fface-d61e-4b8b-8a0a-783123e5c1fe","Type":"ContainerStarted","Data":"de4ac459bc81cb26d5a81ad892aad2de631c698d195ab4b1bbb647c8fb2c58d4"} Nov 02 22:10:19.885898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:19.885841 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:20.126710 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:20.126677 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:20.344499 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:20.344458 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:22.785457 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:22.785422 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-scqp5" event={"ID":"fb6fface-d61e-4b8b-8a0a-783123e5c1fe","Type":"ContainerStarted","Data":"fc295ef687ef48b5287eba3f48dbbf519bab61f13d8a756fd1d302e899208f9f"} Nov 02 22:10:22.785845 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:22.785525 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-scqp5" Nov 02 22:10:22.900012 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:22.899951 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-scqp5" podStartSLOduration=2.843390119 podStartE2EDuration="5.899934682s" totalImagesPullingTime="3.056544563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:22.898467194 +0000 UTC m=+245.710797372" watchObservedRunningTime="2026-11-02 22:10:22.899934682 +0000 UTC m=+245.712264860" Nov 02 22:10:23.814168 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:23.814131 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:24.077382 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:24.077300 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:24.257661 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:24.257626 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:25.065093 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:25.065054 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:25.246534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:25.246500 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:27.761167 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:27.761135 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-d99fd484b-bqftq" Nov 02 22:10:29.506493 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:29.506461 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:29.683641 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:29.683603 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:29.909678 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:29.909646 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:30.099892 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:30.099839 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:31.042635 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:31.042599 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:31.263024 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:31.262973 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:33.792268 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:33.792237 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-scqp5" Nov 02 22:10:36.552156 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:36.552101 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:36.735108 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:36.735074 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:39.499753 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:39.499716 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:39.686144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:39.686099 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:41.345525 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:41.345492 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:41.609869 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:41.609780 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:42.045653 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:42.045591 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:42.241590 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:42.241555 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:43.376950 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.376913 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd"] Nov 02 22:10:43.378322 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.378293 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:43.382305 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.382284 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd" Nov 02 22:10:43.425756 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.425719 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd2067a6-51bb-4d34-ae53-cfd425b26c3b-tmp\") pod \"kube-auth-proxy-6f985b8d7-l5mgd\" (UID: \"cd2067a6-51bb-4d34-ae53-cfd425b26c3b\") " pod="openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd" Nov 02 22:10:43.425945 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.425767 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjv9r\" (UniqueName: \"kubernetes.io/projected/cd2067a6-51bb-4d34-ae53-cfd425b26c3b-kube-api-access-mjv9r\") pod \"kube-auth-proxy-6f985b8d7-l5mgd\" (UID: \"cd2067a6-51bb-4d34-ae53-cfd425b26c3b\") " pod="openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd" Nov 02 22:10:43.425945 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.425924 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd2067a6-51bb-4d34-ae53-cfd425b26c3b-tls-certs\") pod \"kube-auth-proxy-6f985b8d7-l5mgd\" (UID: \"cd2067a6-51bb-4d34-ae53-cfd425b26c3b\") " pod="openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd" Nov 02 22:10:43.473381 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.473349 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 22:10:43.473381 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.473379 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-84m9x\"" Nov 02 22:10:43.477511 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.477487 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 22:10:43.479234 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.479205 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd"] Nov 02 22:10:43.523425 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.523380 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 22:10:43.523647 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.523622 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 22:10:43.527090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.527061 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd2067a6-51bb-4d34-ae53-cfd425b26c3b-tmp\") pod \"kube-auth-proxy-6f985b8d7-l5mgd\" (UID: \"cd2067a6-51bb-4d34-ae53-cfd425b26c3b\") " pod="openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd" Nov 02 22:10:43.527223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.527103 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjv9r\" (UniqueName: \"kubernetes.io/projected/cd2067a6-51bb-4d34-ae53-cfd425b26c3b-kube-api-access-mjv9r\") pod \"kube-auth-proxy-6f985b8d7-l5mgd\" (UID: \"cd2067a6-51bb-4d34-ae53-cfd425b26c3b\") " pod="openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd" Nov 02 22:10:43.527223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.527153 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd2067a6-51bb-4d34-ae53-cfd425b26c3b-tls-certs\") pod \"kube-auth-proxy-6f985b8d7-l5mgd\" (UID: \"cd2067a6-51bb-4d34-ae53-cfd425b26c3b\") " pod="openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd" Nov 02 22:10:43.529391 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.529370 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd2067a6-51bb-4d34-ae53-cfd425b26c3b-tmp\") pod \"kube-auth-proxy-6f985b8d7-l5mgd\" (UID: \"cd2067a6-51bb-4d34-ae53-cfd425b26c3b\") " pod="openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd" Nov 02 22:10:43.529518 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.529500 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd2067a6-51bb-4d34-ae53-cfd425b26c3b-tls-certs\") pod \"kube-auth-proxy-6f985b8d7-l5mgd\" (UID: \"cd2067a6-51bb-4d34-ae53-cfd425b26c3b\") " pod="openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd" Nov 02 22:10:43.566017 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.565986 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:43.584761 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.584724 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjv9r\" (UniqueName: \"kubernetes.io/projected/cd2067a6-51bb-4d34-ae53-cfd425b26c3b-kube-api-access-mjv9r\") pod \"kube-auth-proxy-6f985b8d7-l5mgd\" (UID: \"cd2067a6-51bb-4d34-ae53-cfd425b26c3b\") " pod="openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd" Nov 02 22:10:43.694011 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.693923 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd" Nov 02 22:10:43.776150 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:10:43.776115 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:10:43.952415 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:43.952285 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:44.067509 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:44.067469 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd"] Nov 02 22:10:44.074635 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:10:44.074604 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2067a6_51bb_4d34_ae53_cfd425b26c3b.slice/crio-73877622e987dbc04279750e915dd6a76b565949714205209abb31e75f8ec6e1 WatchSource:0}: Error finding container 73877622e987dbc04279750e915dd6a76b565949714205209abb31e75f8ec6e1: Status 404 returned error can't find the container with id 73877622e987dbc04279750e915dd6a76b565949714205209abb31e75f8ec6e1 Nov 02 22:10:44.137219 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:44.137179 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:44.308291 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:44.308158 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:44.524786 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:44.524754 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:44.704819 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:44.704734 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:44.888913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:44.888856 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd" event={"ID":"cd2067a6-51bb-4d34-ae53-cfd425b26c3b","Type":"ContainerStarted","Data":"73877622e987dbc04279750e915dd6a76b565949714205209abb31e75f8ec6e1"} Nov 02 22:10:45.834391 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:45.834359 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:46.096053 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:46.095960 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:46.293211 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:46.293164 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:47.286341 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:47.286308 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-t28rc"] Nov 02 22:10:47.288794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:47.288772 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-t28rc" Nov 02 22:10:47.347253 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:47.347226 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-t28rc"] Nov 02 22:10:47.364847 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:47.364582 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hg4s\" (UniqueName: \"kubernetes.io/projected/2126c088-3a2e-49a0-b1b7-8736f480cb78-kube-api-access-7hg4s\") pod \"authorino-57f7f8659f-t28rc\" (UID: \"2126c088-3a2e-49a0-b1b7-8736f480cb78\") " pod="kuadrant-system/authorino-57f7f8659f-t28rc" Nov 02 22:10:47.390340 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:47.390317 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-88hkt\"" Nov 02 22:10:47.456682 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:47.456633 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:47.465803 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:47.465770 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hg4s\" (UniqueName: \"kubernetes.io/projected/2126c088-3a2e-49a0-b1b7-8736f480cb78-kube-api-access-7hg4s\") pod \"authorino-57f7f8659f-t28rc\" (UID: \"2126c088-3a2e-49a0-b1b7-8736f480cb78\") " pod="kuadrant-system/authorino-57f7f8659f-t28rc" Nov 02 22:10:47.523725 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:47.523681 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hg4s\" (UniqueName: \"kubernetes.io/projected/2126c088-3a2e-49a0-b1b7-8736f480cb78-kube-api-access-7hg4s\") pod \"authorino-57f7f8659f-t28rc\" (UID: \"2126c088-3a2e-49a0-b1b7-8736f480cb78\") " pod="kuadrant-system/authorino-57f7f8659f-t28rc" Nov 02 22:10:47.602001 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:47.601909 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-t28rc" Nov 02 22:10:47.636722 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:47.636670 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:47.910892 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:47.910833 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd" event={"ID":"cd2067a6-51bb-4d34-ae53-cfd425b26c3b","Type":"ContainerStarted","Data":"22db56cdacacbeed07745fd0819cf4a84d27cdb495bc7ac8a3d4ecfe2d3d5976"} Nov 02 22:10:47.976055 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:47.976018 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-t28rc"] Nov 02 22:10:47.983138 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:10:47.983106 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2126c088_3a2e_49a0_b1b7_8736f480cb78.slice/crio-f750403302593e8d0908f0a1eb5f802f5a900890ee5a2fba1fd34b15ab7c99b4 WatchSource:0}: Error finding container f750403302593e8d0908f0a1eb5f802f5a900890ee5a2fba1fd34b15ab7c99b4: Status 404 returned error can't find the container with id f750403302593e8d0908f0a1eb5f802f5a900890ee5a2fba1fd34b15ab7c99b4 Nov 02 22:10:48.025034 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:48.024967 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6f985b8d7-l5mgd" podStartSLOduration=1.776266109 podStartE2EDuration="5.024946833s" totalImagesPullingTime="3.248680724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:48.023770879 +0000 UTC m=+270.836101056" watchObservedRunningTime="2026-11-02 22:10:48.024946833 +0000 UTC m=+270.837277014" Nov 02 22:10:48.492866 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:48.492825 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:48.671638 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:48.671602 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:48.916856 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:48.916819 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-t28rc" event={"ID":"2126c088-3a2e-49a0-b1b7-8736f480cb78","Type":"ContainerStarted","Data":"f750403302593e8d0908f0a1eb5f802f5a900890ee5a2fba1fd34b15ab7c99b4"} Nov 02 22:10:49.940544 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:49.940504 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:10:49.944120 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:49.944096 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-gzdhm" Nov 02 22:10:49.989481 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:49.989437 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g7tc\" (UniqueName: \"kubernetes.io/projected/6f279b88-919c-4c3d-a46c-f5af9b1fbd63-kube-api-access-9g7tc\") pod \"limitador-limitador-7c9b87d574-gzdhm\" (UID: \"6f279b88-919c-4c3d-a46c-f5af9b1fbd63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gzdhm" Nov 02 22:10:49.989481 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:49.989481 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6f279b88-919c-4c3d-a46c-f5af9b1fbd63-config-file\") pod \"limitador-limitador-7c9b87d574-gzdhm\" (UID: \"6f279b88-919c-4c3d-a46c-f5af9b1fbd63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gzdhm" Nov 02 22:10:49.995645 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:49.995600 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 22:10:49.997847 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:49.997820 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:10:50.042043 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:50.042011 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qkcwg\"" Nov 02 22:10:50.090792 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:50.090748 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g7tc\" (UniqueName: \"kubernetes.io/projected/6f279b88-919c-4c3d-a46c-f5af9b1fbd63-kube-api-access-9g7tc\") pod \"limitador-limitador-7c9b87d574-gzdhm\" (UID: \"6f279b88-919c-4c3d-a46c-f5af9b1fbd63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gzdhm" Nov 02 22:10:50.090792 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:50.090797 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6f279b88-919c-4c3d-a46c-f5af9b1fbd63-config-file\") pod \"limitador-limitador-7c9b87d574-gzdhm\" (UID: \"6f279b88-919c-4c3d-a46c-f5af9b1fbd63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gzdhm" Nov 02 22:10:50.091487 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:50.091465 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6f279b88-919c-4c3d-a46c-f5af9b1fbd63-config-file\") pod \"limitador-limitador-7c9b87d574-gzdhm\" (UID: \"6f279b88-919c-4c3d-a46c-f5af9b1fbd63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gzdhm" Nov 02 22:10:50.148943 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:50.148899 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g7tc\" (UniqueName: \"kubernetes.io/projected/6f279b88-919c-4c3d-a46c-f5af9b1fbd63-kube-api-access-9g7tc\") pod \"limitador-limitador-7c9b87d574-gzdhm\" (UID: \"6f279b88-919c-4c3d-a46c-f5af9b1fbd63\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gzdhm" Nov 02 22:10:50.231658 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:50.231576 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:10:50.256769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:50.256728 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-gzdhm" Nov 02 22:10:50.587739 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:50.587652 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:50.636084 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:50.636048 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:10:50.643055 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:10:50.643003 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f279b88_919c_4c3d_a46c_f5af9b1fbd63.slice/crio-b763be435b11cf00f9bc1808b2f902e72d703868d192edba0dc4b54c1ba81a83 WatchSource:0}: Error finding container b763be435b11cf00f9bc1808b2f902e72d703868d192edba0dc4b54c1ba81a83: Status 404 returned error can't find the container with id b763be435b11cf00f9bc1808b2f902e72d703868d192edba0dc4b54c1ba81a83 Nov 02 22:10:50.938010 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:50.937974 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-gzdhm" event={"ID":"6f279b88-919c-4c3d-a46c-f5af9b1fbd63","Type":"ContainerStarted","Data":"b763be435b11cf00f9bc1808b2f902e72d703868d192edba0dc4b54c1ba81a83"} Nov 02 22:10:51.946784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:51.946738 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-t28rc" event={"ID":"2126c088-3a2e-49a0-b1b7-8736f480cb78","Type":"ContainerStarted","Data":"7968ff8b19d97d8126664094a9b1608e8a1ba757aba81ab1ab615c2839782a2c"} Nov 02 22:10:52.064579 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:52.064522 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-t28rc" podStartSLOduration=1.7330635810000001 podStartE2EDuration="5.064506039s" totalImagesPullingTime="3.331442458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:52.062770959 +0000 UTC m=+274.875101139" watchObservedRunningTime="2026-11-02 22:10:52.064506039 +0000 UTC m=+274.876836217" Nov 02 22:10:52.953698 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:52.953649 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-gzdhm" event={"ID":"6f279b88-919c-4c3d-a46c-f5af9b1fbd63","Type":"ContainerStarted","Data":"9faff3b7050c9c52039bbf56506c1c8603ef47f8749298e660e1e255f715540f"} Nov 02 22:10:52.954169 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:52.953827 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-gzdhm" Nov 02 22:10:53.063015 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:53.062855 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-gzdhm" podStartSLOduration=1.988831825 podStartE2EDuration="4.06283484s" totalImagesPullingTime="2.074003015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:53.062078805 +0000 UTC m=+275.874408987" watchObservedRunningTime="2026-11-02 22:10:53.06283484 +0000 UTC m=+275.875165016" Nov 02 22:10:53.536407 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:53.536374 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-t28rc"] Nov 02 22:10:53.958785 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:53.958747 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-t28rc" podUID="2126c088-3a2e-49a0-b1b7-8736f480cb78" containerName="authorino" containerID="cri-o://7968ff8b19d97d8126664094a9b1608e8a1ba757aba81ab1ab615c2839782a2c" gracePeriod=30 Nov 02 22:10:54.206792 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:54.206455 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-t28rc" Nov 02 22:10:54.331040 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:54.330957 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2126c088-3a2e-49a0-b1b7-8736f480cb78-kube-api-access-7hg4s\" (UniqueName: \"kubernetes.io/projected/2126c088-3a2e-49a0-b1b7-8736f480cb78-kube-api-access-7hg4s\") pod \"2126c088-3a2e-49a0-b1b7-8736f480cb78\" (UID: \"2126c088-3a2e-49a0-b1b7-8736f480cb78\") " Nov 02 22:10:54.333132 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:54.333095 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2126c088-3a2e-49a0-b1b7-8736f480cb78-kube-api-access-7hg4s" pod "2126c088-3a2e-49a0-b1b7-8736f480cb78" (UID: "2126c088-3a2e-49a0-b1b7-8736f480cb78"). InnerVolumeSpecName "kube-api-access-7hg4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:54.431710 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:54.431632 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hg4s\" (UniqueName: \"kubernetes.io/projected/2126c088-3a2e-49a0-b1b7-8736f480cb78-kube-api-access-7hg4s\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:54.965091 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:54.965054 5871 generic.go:356] "Generic (PLEG): container finished" podID="2126c088-3a2e-49a0-b1b7-8736f480cb78" containerID="7968ff8b19d97d8126664094a9b1608e8a1ba757aba81ab1ab615c2839782a2c" exitCode=0 Nov 02 22:10:54.965613 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:54.965104 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-t28rc" Nov 02 22:10:54.965613 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:54.965149 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-t28rc" event={"ID":"2126c088-3a2e-49a0-b1b7-8736f480cb78","Type":"ContainerDied","Data":"7968ff8b19d97d8126664094a9b1608e8a1ba757aba81ab1ab615c2839782a2c"} Nov 02 22:10:54.965613 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:54.965195 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-t28rc" event={"ID":"2126c088-3a2e-49a0-b1b7-8736f480cb78","Type":"ContainerDied","Data":"f750403302593e8d0908f0a1eb5f802f5a900890ee5a2fba1fd34b15ab7c99b4"} Nov 02 22:10:54.965613 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:54.965212 5871 scope.go:122] "RemoveContainer" containerID="7968ff8b19d97d8126664094a9b1608e8a1ba757aba81ab1ab615c2839782a2c" Nov 02 22:10:54.974740 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:54.974723 5871 scope.go:122] "RemoveContainer" containerID="7968ff8b19d97d8126664094a9b1608e8a1ba757aba81ab1ab615c2839782a2c" Nov 02 22:10:54.975091 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:10:54.975070 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7968ff8b19d97d8126664094a9b1608e8a1ba757aba81ab1ab615c2839782a2c\": container with ID starting with 7968ff8b19d97d8126664094a9b1608e8a1ba757aba81ab1ab615c2839782a2c not found: ID does not exist" containerID="7968ff8b19d97d8126664094a9b1608e8a1ba757aba81ab1ab615c2839782a2c" Nov 02 22:10:54.975160 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:54.975105 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7968ff8b19d97d8126664094a9b1608e8a1ba757aba81ab1ab615c2839782a2c"} err="failed to get container status \"7968ff8b19d97d8126664094a9b1608e8a1ba757aba81ab1ab615c2839782a2c\": rpc error: code = NotFound desc = could not find container \"7968ff8b19d97d8126664094a9b1608e8a1ba757aba81ab1ab615c2839782a2c\": container with ID starting with 7968ff8b19d97d8126664094a9b1608e8a1ba757aba81ab1ab615c2839782a2c not found: ID does not exist" Nov 02 22:10:55.120193 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:55.120158 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-t28rc"] Nov 02 22:10:55.124145 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:55.124115 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-t28rc"] Nov 02 22:10:55.311825 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:55.311746 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2126c088-3a2e-49a0-b1b7-8736f480cb78" path="/var/lib/kubelet/pods/2126c088-3a2e-49a0-b1b7-8736f480cb78/volumes" Nov 02 22:10:55.540606 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:55.540570 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:55.723024 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:55.722993 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:58.486364 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:58.486323 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:10:58.666448 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:10:58.666395 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:01.925122 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:01.925091 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:02.106993 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:02.106959 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:03.959720 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:03.959688 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-gzdhm" Nov 02 22:11:07.737518 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:07.737472 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:07.910461 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:07.910426 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:08.086158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:08.086064 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:09.112331 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:09.112292 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:09.297966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:09.297924 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:09.908527 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:09.908493 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:13.769710 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:11:13.769670 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:11:14.789804 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:14.789775 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:14.982617 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:14.982578 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:19.572668 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:19.572631 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:24.138236 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:24.138190 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:31.593153 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:31.593106 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:31.780100 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:31.780057 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:31.951109 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:31.951067 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:32.137379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:32.137344 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:35.915375 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:35.915341 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:36.092331 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:36.092300 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:38.869112 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:38.869076 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9"] Nov 02 22:11:38.869524 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:38.869466 5871 cpu_manager.go:414] "containerMap: removing container" podUID="2126c088-3a2e-49a0-b1b7-8736f480cb78" containerName="authorino" Nov 02 22:11:38.869524 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:38.869478 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2126c088-3a2e-49a0-b1b7-8736f480cb78" containerName="authorino" Nov 02 22:11:38.869594 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:38.869545 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="2126c088-3a2e-49a0-b1b7-8736f480cb78" containerName="authorino" Nov 02 22:11:38.872577 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:38.872555 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:38.921244 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:38.921208 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 22:11:38.921244 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:38.921243 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 22:11:38.921460 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:38.921449 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 22:11:38.921500 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:38.921449 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 22:11:38.923884 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:38.923846 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/954286ee-b7ba-4a67-9c30-904f75d92433-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqst9\" (UID: \"954286ee-b7ba-4a67-9c30-904f75d92433\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:38.924006 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:38.923907 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt9cw\" (UniqueName: \"kubernetes.io/projected/954286ee-b7ba-4a67-9c30-904f75d92433-kube-api-access-mt9cw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqst9\" (UID: \"954286ee-b7ba-4a67-9c30-904f75d92433\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:38.924006 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:38.923977 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/954286ee-b7ba-4a67-9c30-904f75d92433-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqst9\" (UID: \"954286ee-b7ba-4a67-9c30-904f75d92433\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:38.924091 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:38.924040 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/954286ee-b7ba-4a67-9c30-904f75d92433-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqst9\" (UID: \"954286ee-b7ba-4a67-9c30-904f75d92433\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:38.967691 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:38.967655 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 22:11:38.967904 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:38.967737 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-nh2bl\"" Nov 02 22:11:38.969754 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:38.969727 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9"] Nov 02 22:11:39.025431 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:39.025398 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt9cw\" (UniqueName: \"kubernetes.io/projected/954286ee-b7ba-4a67-9c30-904f75d92433-kube-api-access-mt9cw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqst9\" (UID: \"954286ee-b7ba-4a67-9c30-904f75d92433\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:39.025431 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:39.025431 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/954286ee-b7ba-4a67-9c30-904f75d92433-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqst9\" (UID: \"954286ee-b7ba-4a67-9c30-904f75d92433\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:39.025651 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:39.025475 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/954286ee-b7ba-4a67-9c30-904f75d92433-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqst9\" (UID: \"954286ee-b7ba-4a67-9c30-904f75d92433\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:39.025651 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:39.025525 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/954286ee-b7ba-4a67-9c30-904f75d92433-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqst9\" (UID: \"954286ee-b7ba-4a67-9c30-904f75d92433\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:39.025651 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:11:39.025601 5871 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 02 22:11:39.025774 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:11:39.025695 5871 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/954286ee-b7ba-4a67-9c30-904f75d92433-proxy-source podName:954286ee-b7ba-4a67-9c30-904f75d92433 nodeName:}" failed. No retries permitted until 2026-11-02 22:11:39.525672146 +0000 UTC m=+322.338002316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/954286ee-b7ba-4a67-9c30-904f75d92433-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-kqst9" (UID: "954286ee-b7ba-4a67-9c30-904f75d92433") : configmap "usage-tenancy-proxy-source" not found Nov 02 22:11:39.025774 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:11:39.025713 5871 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 02 22:11:39.025851 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:11:39.025784 5871 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/954286ee-b7ba-4a67-9c30-904f75d92433-tls-cert podName:954286ee-b7ba-4a67-9c30-904f75d92433 nodeName:}" failed. No retries permitted until 2026-11-02 22:11:39.525765065 +0000 UTC m=+322.338095239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/954286ee-b7ba-4a67-9c30-904f75d92433-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-kqst9" (UID: "954286ee-b7ba-4a67-9c30-904f75d92433") : secret "usage-logs-tenancy-proxy-tls" not found Nov 02 22:11:39.026312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:39.026293 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/954286ee-b7ba-4a67-9c30-904f75d92433-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqst9\" (UID: \"954286ee-b7ba-4a67-9c30-904f75d92433\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:39.063393 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:39.063352 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:39.083172 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:39.083131 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt9cw\" (UniqueName: \"kubernetes.io/projected/954286ee-b7ba-4a67-9c30-904f75d92433-kube-api-access-mt9cw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqst9\" (UID: \"954286ee-b7ba-4a67-9c30-904f75d92433\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:39.247507 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:39.247374 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:11:39.530763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:39.530676 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/954286ee-b7ba-4a67-9c30-904f75d92433-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqst9\" (UID: \"954286ee-b7ba-4a67-9c30-904f75d92433\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:39.530763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:39.530737 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/954286ee-b7ba-4a67-9c30-904f75d92433-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqst9\" (UID: \"954286ee-b7ba-4a67-9c30-904f75d92433\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:39.531572 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:39.531542 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/954286ee-b7ba-4a67-9c30-904f75d92433-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqst9\" (UID: \"954286ee-b7ba-4a67-9c30-904f75d92433\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:39.533241 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:39.533218 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/954286ee-b7ba-4a67-9c30-904f75d92433-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqst9\" (UID: \"954286ee-b7ba-4a67-9c30-904f75d92433\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:39.785193 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:39.785100 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:40.160204 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:40.160170 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9"] Nov 02 22:11:40.167429 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:11:40.167396 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954286ee_b7ba_4a67_9c30_904f75d92433.slice/crio-f2ac76d3cf582791da30a09b6537c83d21b4c373753dca23ca76aa98aea0045e WatchSource:0}: Error finding container f2ac76d3cf582791da30a09b6537c83d21b4c373753dca23ca76aa98aea0045e: Status 404 returned error can't find the container with id f2ac76d3cf582791da30a09b6537c83d21b4c373753dca23ca76aa98aea0045e Nov 02 22:11:40.189683 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:40.189643 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" event={"ID":"954286ee-b7ba-4a67-9c30-904f75d92433","Type":"ContainerStarted","Data":"f2ac76d3cf582791da30a09b6537c83d21b4c373753dca23ca76aa98aea0045e"} Nov 02 22:11:43.777122 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:11:43.777081 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:11:53.274149 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:53.274047 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" event={"ID":"954286ee-b7ba-4a67-9c30-904f75d92433","Type":"ContainerStarted","Data":"dd28dc917e77c8929fee4d3b2003cb942fc281eba565bc0386edc0ab052282fc"} Nov 02 22:11:53.274560 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:53.274144 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:11:53.385173 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:11:53.385103 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" podStartSLOduration=2.572446763 podStartE2EDuration="15.385088041s" totalImagesPullingTime="12.812641278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:53.383794828 +0000 UTC m=+336.196125006" watchObservedRunningTime="2026-11-02 22:11:53.385088041 +0000 UTC m=+336.197418218" Nov 02 22:12:03.662290 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:03.662212 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:03.928110 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:03.928025 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:04.285183 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:04.285095 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqst9" Nov 02 22:12:05.879303 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:05.879259 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:06.105653 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:06.105610 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:06.282184 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:06.282090 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:10.966980 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:10.966940 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:12:13.770791 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:12:13.770750 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:12:14.419465 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:14.419433 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:14.638927 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:14.638854 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:14.886148 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:14.886104 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:15.071684 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.071652 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:15.252908 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.252795 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:15.432443 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.432413 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:15.513902 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.513797 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-mn4sc"] Nov 02 22:12:15.520182 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.520153 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-mn4sc" Nov 02 22:12:15.571355 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.571307 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6g8v\" (UniqueName: \"kubernetes.io/projected/28482f90-219d-4eae-b413-f5413c9ef08b-kube-api-access-v6g8v\") pod \"odh-model-controller-769875db56-mn4sc\" (UID: \"28482f90-219d-4eae-b413-f5413c9ef08b\") " pod="opendatahub/odh-model-controller-769875db56-mn4sc" Nov 02 22:12:15.571543 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.571370 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28482f90-219d-4eae-b413-f5413c9ef08b-metrics-certs\") pod \"odh-model-controller-769875db56-mn4sc\" (UID: \"28482f90-219d-4eae-b413-f5413c9ef08b\") " pod="opendatahub/odh-model-controller-769875db56-mn4sc" Nov 02 22:12:15.571664 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.571590 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28482f90-219d-4eae-b413-f5413c9ef08b-cert\") pod \"odh-model-controller-769875db56-mn4sc\" (UID: \"28482f90-219d-4eae-b413-f5413c9ef08b\") " pod="opendatahub/odh-model-controller-769875db56-mn4sc" Nov 02 22:12:15.572263 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.572240 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-mn4sc"] Nov 02 22:12:15.616084 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.616054 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-kjpfx\"" Nov 02 22:12:15.616084 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.616071 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 22:12:15.616308 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.616108 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 22:12:15.672650 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.672602 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28482f90-219d-4eae-b413-f5413c9ef08b-metrics-certs\") pod \"odh-model-controller-769875db56-mn4sc\" (UID: \"28482f90-219d-4eae-b413-f5413c9ef08b\") " pod="opendatahub/odh-model-controller-769875db56-mn4sc" Nov 02 22:12:15.672836 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.672699 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28482f90-219d-4eae-b413-f5413c9ef08b-cert\") pod \"odh-model-controller-769875db56-mn4sc\" (UID: \"28482f90-219d-4eae-b413-f5413c9ef08b\") " pod="opendatahub/odh-model-controller-769875db56-mn4sc" Nov 02 22:12:15.672836 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.672743 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6g8v\" (UniqueName: \"kubernetes.io/projected/28482f90-219d-4eae-b413-f5413c9ef08b-kube-api-access-v6g8v\") pod \"odh-model-controller-769875db56-mn4sc\" (UID: \"28482f90-219d-4eae-b413-f5413c9ef08b\") " pod="opendatahub/odh-model-controller-769875db56-mn4sc" Nov 02 22:12:15.675504 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.675476 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28482f90-219d-4eae-b413-f5413c9ef08b-cert\") pod \"odh-model-controller-769875db56-mn4sc\" (UID: \"28482f90-219d-4eae-b413-f5413c9ef08b\") " pod="opendatahub/odh-model-controller-769875db56-mn4sc" Nov 02 22:12:15.675639 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.675617 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28482f90-219d-4eae-b413-f5413c9ef08b-metrics-certs\") pod \"odh-model-controller-769875db56-mn4sc\" (UID: \"28482f90-219d-4eae-b413-f5413c9ef08b\") " pod="opendatahub/odh-model-controller-769875db56-mn4sc" Nov 02 22:12:15.740620 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.740588 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6g8v\" (UniqueName: \"kubernetes.io/projected/28482f90-219d-4eae-b413-f5413c9ef08b-kube-api-access-v6g8v\") pod \"odh-model-controller-769875db56-mn4sc\" (UID: \"28482f90-219d-4eae-b413-f5413c9ef08b\") " pod="opendatahub/odh-model-controller-769875db56-mn4sc" Nov 02 22:12:15.833411 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:15.833325 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-mn4sc" Nov 02 22:12:16.205152 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:16.205121 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-mn4sc"] Nov 02 22:12:16.213392 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:12:16.213349 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28482f90_219d_4eae_b413_f5413c9ef08b.slice/crio-7fa0ac7bf97e6420a55952e707e218f640a64ff5c756487510effca4e7766967 WatchSource:0}: Error finding container 7fa0ac7bf97e6420a55952e707e218f640a64ff5c756487510effca4e7766967: Status 404 returned error can't find the container with id 7fa0ac7bf97e6420a55952e707e218f640a64ff5c756487510effca4e7766967 Nov 02 22:12:16.388823 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:16.388787 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-mn4sc" event={"ID":"28482f90-219d-4eae-b413-f5413c9ef08b","Type":"ContainerStarted","Data":"7fa0ac7bf97e6420a55952e707e218f640a64ff5c756487510effca4e7766967"} Nov 02 22:12:16.525225 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:16.525134 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:19.868718 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:19.868607 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:20.410846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:20.410810 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-mn4sc" event={"ID":"28482f90-219d-4eae-b413-f5413c9ef08b","Type":"ContainerStarted","Data":"58d4e2e899877ead229122e47711b90f59bd45a7c005878422ec6796b312c34e"} Nov 02 22:12:20.411047 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:20.410945 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-mn4sc" Nov 02 22:12:20.522940 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:20.522860 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-mn4sc" podStartSLOduration=2.097100134 podStartE2EDuration="5.522844973s" totalImagesPullingTime="3.425744839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:20.521532183 +0000 UTC m=+363.333862360" watchObservedRunningTime="2026-11-02 22:12:20.522844973 +0000 UTC m=+363.335175152" Nov 02 22:12:21.364889 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:21.364832 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:21.542745 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:21.542705 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:21.729235 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:21.729143 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:21.908096 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:21.908059 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:22.094865 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:22.094783 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:22.274041 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:22.274002 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:22.445647 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:22.445615 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:24.495253 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:24.495160 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:27.369258 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.369218 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:27.639787 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.639696 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-mxzfk"] Nov 02 22:12:27.642447 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.642424 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-mxzfk" Nov 02 22:12:27.679450 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.679403 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trgwg\" (UniqueName: \"kubernetes.io/projected/dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d-kube-api-access-trgwg\") pod \"authorino-6f6944bcdd-mxzfk\" (UID: \"dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d\") " pod="kuadrant-system/authorino-6f6944bcdd-mxzfk" Nov 02 22:12:27.691495 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.691463 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-88hkt\"" Nov 02 22:12:27.707533 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.707502 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:27.739464 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.739427 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-mxzfk"] Nov 02 22:12:27.780249 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.780210 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trgwg\" (UniqueName: \"kubernetes.io/projected/dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d-kube-api-access-trgwg\") pod \"authorino-6f6944bcdd-mxzfk\" (UID: \"dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d\") " pod="kuadrant-system/authorino-6f6944bcdd-mxzfk" Nov 02 22:12:27.828900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.828852 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bb6546495-c4r29"] Nov 02 22:12:27.831783 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.831758 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:12:27.845849 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.845815 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trgwg\" (UniqueName: \"kubernetes.io/projected/dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d-kube-api-access-trgwg\") pod \"authorino-6f6944bcdd-mxzfk\" (UID: \"dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d\") " pod="kuadrant-system/authorino-6f6944bcdd-mxzfk" Nov 02 22:12:27.880791 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.880750 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdbnq\" (UniqueName: \"kubernetes.io/projected/6bcc67dd-503f-415d-a5b5-6ee333124562-kube-api-access-rdbnq\") pod \"maas-api-5bb6546495-c4r29\" (UID: \"6bcc67dd-503f-415d-a5b5-6ee333124562\") " pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:12:27.881035 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.880867 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bcc67dd-503f-415d-a5b5-6ee333124562-maas-api-tls\") pod \"maas-api-5bb6546495-c4r29\" (UID: \"6bcc67dd-503f-415d-a5b5-6ee333124562\") " pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:12:27.881035 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.880959 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:12:27.881035 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.880961 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bcc67dd-503f-415d-a5b5-6ee333124562-metrics-certs\") pod \"maas-api-5bb6546495-c4r29\" (UID: \"6bcc67dd-503f-415d-a5b5-6ee333124562\") " pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:12:27.881035 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.880970 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l4djg\"" Nov 02 22:12:27.881035 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.880971 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:12:27.881319 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.881043 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 22:12:27.881319 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.881073 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bcc67dd-503f-415d-a5b5-6ee333124562-openshift-service-ca\") pod \"maas-api-5bb6546495-c4r29\" (UID: \"6bcc67dd-503f-415d-a5b5-6ee333124562\") " pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:12:27.881319 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.881108 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 22:12:27.889172 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.889129 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bb6546495-c4r29"] Nov 02 22:12:27.953425 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.953335 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-mxzfk" Nov 02 22:12:27.982001 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.981965 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdbnq\" (UniqueName: \"kubernetes.io/projected/6bcc67dd-503f-415d-a5b5-6ee333124562-kube-api-access-rdbnq\") pod \"maas-api-5bb6546495-c4r29\" (UID: \"6bcc67dd-503f-415d-a5b5-6ee333124562\") " pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:12:27.982174 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.982013 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bcc67dd-503f-415d-a5b5-6ee333124562-maas-api-tls\") pod \"maas-api-5bb6546495-c4r29\" (UID: \"6bcc67dd-503f-415d-a5b5-6ee333124562\") " pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:12:27.982174 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.982048 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bcc67dd-503f-415d-a5b5-6ee333124562-metrics-certs\") pod \"maas-api-5bb6546495-c4r29\" (UID: \"6bcc67dd-503f-415d-a5b5-6ee333124562\") " pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:12:27.982174 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.982079 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bcc67dd-503f-415d-a5b5-6ee333124562-openshift-service-ca\") pod \"maas-api-5bb6546495-c4r29\" (UID: \"6bcc67dd-503f-415d-a5b5-6ee333124562\") " pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:12:27.982920 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.982864 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bcc67dd-503f-415d-a5b5-6ee333124562-openshift-service-ca\") pod \"maas-api-5bb6546495-c4r29\" (UID: \"6bcc67dd-503f-415d-a5b5-6ee333124562\") " pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:12:27.984609 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.984587 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bcc67dd-503f-415d-a5b5-6ee333124562-metrics-certs\") pod \"maas-api-5bb6546495-c4r29\" (UID: \"6bcc67dd-503f-415d-a5b5-6ee333124562\") " pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:12:27.985372 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.985352 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bcc67dd-503f-415d-a5b5-6ee333124562-maas-api-tls\") pod \"maas-api-5bb6546495-c4r29\" (UID: \"6bcc67dd-503f-415d-a5b5-6ee333124562\") " pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:12:27.999249 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:27.999214 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-mxzfk"] Nov 02 22:12:28.040839 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:28.040807 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdbnq\" (UniqueName: \"kubernetes.io/projected/6bcc67dd-503f-415d-a5b5-6ee333124562-kube-api-access-rdbnq\") pod \"maas-api-5bb6546495-c4r29\" (UID: \"6bcc67dd-503f-415d-a5b5-6ee333124562\") " pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:12:28.143952 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:28.143912 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:12:28.324355 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:28.324270 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-mxzfk"] Nov 02 22:12:28.331619 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:12:28.331585 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd4f06c4_5e49_4ae5_a27b_ea6dc418c89d.slice/crio-acd3b9f0414bb006045ae110ab8ae1f74a38971886b69b9bf481c3b4694307fa WatchSource:0}: Error finding container acd3b9f0414bb006045ae110ab8ae1f74a38971886b69b9bf481c3b4694307fa: Status 404 returned error can't find the container with id acd3b9f0414bb006045ae110ab8ae1f74a38971886b69b9bf481c3b4694307fa Nov 02 22:12:28.452043 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:28.452002 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-mxzfk" event={"ID":"dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d","Type":"ContainerStarted","Data":"acd3b9f0414bb006045ae110ab8ae1f74a38971886b69b9bf481c3b4694307fa"} Nov 02 22:12:28.519111 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:28.519071 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bb6546495-c4r29"] Nov 02 22:12:28.527180 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:12:28.527146 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bcc67dd_503f_415d_a5b5_6ee333124562.slice/crio-bc90ab77d4152dca88930d767793194729be79b4af7641ea9d09820b32344509 WatchSource:0}: Error finding container bc90ab77d4152dca88930d767793194729be79b4af7641ea9d09820b32344509: Status 404 returned error can't find the container with id bc90ab77d4152dca88930d767793194729be79b4af7641ea9d09820b32344509 Nov 02 22:12:29.459060 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:29.459023 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" event={"ID":"6bcc67dd-503f-415d-a5b5-6ee333124562","Type":"ContainerStarted","Data":"bc90ab77d4152dca88930d767793194729be79b4af7641ea9d09820b32344509"} Nov 02 22:12:30.468725 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:30.468681 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-mxzfk" event={"ID":"dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d","Type":"ContainerStarted","Data":"5703dcec29cc7a8ba5aff9bd424f54310c49a7e5878255df1303703928cbb9d2"} Nov 02 22:12:30.469322 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:30.468771 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-mxzfk" podUID="dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d" containerName="authorino" containerID="cri-o://5703dcec29cc7a8ba5aff9bd424f54310c49a7e5878255df1303703928cbb9d2" gracePeriod=30 Nov 02 22:12:30.581184 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:30.581117 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-mxzfk" podStartSLOduration=2.032240728 podStartE2EDuration="3.581103933s" totalImagesPullingTime="1.548863205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:30.580152547 +0000 UTC m=+373.392482727" watchObservedRunningTime="2026-11-02 22:12:30.581103933 +0000 UTC m=+373.393434111" Nov 02 22:12:30.778825 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:30.778789 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-mxzfk" Nov 02 22:12:30.912083 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:30.912040 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d-kube-api-access-trgwg\" (UniqueName: \"kubernetes.io/projected/dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d-kube-api-access-trgwg\") pod \"dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d\" (UID: \"dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d\") " Nov 02 22:12:30.914559 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:30.914521 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d-kube-api-access-trgwg" pod "dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d" (UID: "dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d"). InnerVolumeSpecName "kube-api-access-trgwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:31.012977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:31.012866 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trgwg\" (UniqueName: \"kubernetes.io/projected/dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d-kube-api-access-trgwg\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:31.419589 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:31.419553 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-mn4sc" Nov 02 22:12:31.487010 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:31.486968 5871 generic.go:356] "Generic (PLEG): container finished" podID="dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d" containerID="5703dcec29cc7a8ba5aff9bd424f54310c49a7e5878255df1303703928cbb9d2" exitCode=0 Nov 02 22:12:31.487500 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:31.487019 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-mxzfk" Nov 02 22:12:31.487500 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:31.487050 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-mxzfk" event={"ID":"dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d","Type":"ContainerDied","Data":"5703dcec29cc7a8ba5aff9bd424f54310c49a7e5878255df1303703928cbb9d2"} Nov 02 22:12:31.487500 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:31.487101 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-mxzfk" event={"ID":"dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d","Type":"ContainerDied","Data":"acd3b9f0414bb006045ae110ab8ae1f74a38971886b69b9bf481c3b4694307fa"} Nov 02 22:12:31.487500 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:31.487125 5871 scope.go:122] "RemoveContainer" containerID="5703dcec29cc7a8ba5aff9bd424f54310c49a7e5878255df1303703928cbb9d2" Nov 02 22:12:31.497308 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:31.497281 5871 scope.go:122] "RemoveContainer" containerID="5703dcec29cc7a8ba5aff9bd424f54310c49a7e5878255df1303703928cbb9d2" Nov 02 22:12:31.500123 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:12:31.500092 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5703dcec29cc7a8ba5aff9bd424f54310c49a7e5878255df1303703928cbb9d2\": container with ID starting with 5703dcec29cc7a8ba5aff9bd424f54310c49a7e5878255df1303703928cbb9d2 not found: ID does not exist" containerID="5703dcec29cc7a8ba5aff9bd424f54310c49a7e5878255df1303703928cbb9d2" Nov 02 22:12:31.500269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:31.500136 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5703dcec29cc7a8ba5aff9bd424f54310c49a7e5878255df1303703928cbb9d2"} err="failed to get container status \"5703dcec29cc7a8ba5aff9bd424f54310c49a7e5878255df1303703928cbb9d2\": rpc error: code = NotFound desc = could not find container \"5703dcec29cc7a8ba5aff9bd424f54310c49a7e5878255df1303703928cbb9d2\": container with ID starting with 5703dcec29cc7a8ba5aff9bd424f54310c49a7e5878255df1303703928cbb9d2 not found: ID does not exist" Nov 02 22:12:31.634939 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:31.634910 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-mxzfk"] Nov 02 22:12:31.637158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:31.637134 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-mxzfk"] Nov 02 22:12:32.493379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:32.493340 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" event={"ID":"6bcc67dd-503f-415d-a5b5-6ee333124562","Type":"ContainerStarted","Data":"158feb877db394c5c4696642f24e5a0aeaab184de3b7ff54786fbf3f53bd2167"} Nov 02 22:12:32.493832 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:32.493495 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:12:32.607369 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:32.607303 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" podStartSLOduration=2.501517861 podStartE2EDuration="5.607288101s" totalImagesPullingTime="3.10577024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:32.604914267 +0000 UTC m=+375.417244445" watchObservedRunningTime="2026-11-02 22:12:32.607288101 +0000 UTC m=+375.419618293" Nov 02 22:12:33.312232 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:33.312192 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d" path="/var/lib/kubelet/pods/dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d/volumes" Nov 02 22:12:34.573572 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:34.573535 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:12:38.505293 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:12:38.505262 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:12:43.770109 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:12:43.770067 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:13:05.214645 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:13:05.214608 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:05.384186 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:13:05.384153 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:07.229231 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:13:07.229196 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:07.410732 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:13:07.410689 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:13.770156 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:13:13.770112 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:13:20.220625 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:13:20.220582 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:34.914397 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:13:34.914361 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:13:43.770362 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:13:43.770320 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:14:06.264846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:06.264751 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:06.449437 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:06.449401 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:13.770780 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:14:13.770738 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:14:19.935161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:19.935120 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:20.575589 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:20.575551 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:24.288655 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:24.288622 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:27.457924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:27.457887 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:14:28.578607 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.578567 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-8qtld"] Nov 02 22:14:28.579031 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.578997 5871 cpu_manager.go:414] "containerMap: removing container" podUID="dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d" containerName="authorino" Nov 02 22:14:28.579031 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.579009 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d" containerName="authorino" Nov 02 22:14:28.579106 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.579090 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd4f06c4-5e49-4ae5-a27b-ea6dc418c89d" containerName="authorino" Nov 02 22:14:28.582274 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.582249 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.631429 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.631395 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 22:14:28.652973 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.652941 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4bfaefa-f570-4bf9-abd6-45231654dafe-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.653149 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.652996 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4bfaefa-f570-4bf9-abd6-45231654dafe-home\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.653205 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.653131 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4bfaefa-f570-4bf9-abd6-45231654dafe-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.653240 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.653204 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsfgt\" (UniqueName: \"kubernetes.io/projected/a4bfaefa-f570-4bf9-abd6-45231654dafe-kube-api-access-gsfgt\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.653274 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.653260 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4bfaefa-f570-4bf9-abd6-45231654dafe-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.653315 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.653299 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4bfaefa-f570-4bf9-abd6-45231654dafe-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.677555 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.677520 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 22:14:28.677719 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.677538 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 02 22:14:28.677719 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.677538 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-swvkh\"" Nov 02 22:14:28.679637 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.679613 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-8qtld"] Nov 02 22:14:28.710690 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.710654 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:28.754514 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.754476 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4bfaefa-f570-4bf9-abd6-45231654dafe-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.754514 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.754521 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsfgt\" (UniqueName: \"kubernetes.io/projected/a4bfaefa-f570-4bf9-abd6-45231654dafe-kube-api-access-gsfgt\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.754822 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.754560 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4bfaefa-f570-4bf9-abd6-45231654dafe-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.754822 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.754595 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4bfaefa-f570-4bf9-abd6-45231654dafe-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.754822 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.754622 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4bfaefa-f570-4bf9-abd6-45231654dafe-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.754822 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.754653 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4bfaefa-f570-4bf9-abd6-45231654dafe-home\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.755071 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.755016 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4bfaefa-f570-4bf9-abd6-45231654dafe-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.755229 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.755209 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4bfaefa-f570-4bf9-abd6-45231654dafe-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.755302 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.755248 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4bfaefa-f570-4bf9-abd6-45231654dafe-home\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.756814 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.756796 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4bfaefa-f570-4bf9-abd6-45231654dafe-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.757176 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.757153 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4bfaefa-f570-4bf9-abd6-45231654dafe-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.812763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.812725 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsfgt\" (UniqueName: \"kubernetes.io/projected/a4bfaefa-f570-4bf9-abd6-45231654dafe-kube-api-access-gsfgt\") pod \"echo-simulator-kserve-64b6f8f96c-8qtld\" (UID: \"a4bfaefa-f570-4bf9-abd6-45231654dafe\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:28.895382 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:28.895352 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:29.267445 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:29.267361 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-8qtld"] Nov 02 22:14:29.274476 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:14:29.274446 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bfaefa_f570_4bf9_abd6_45231654dafe.slice/crio-b4031663aebc38123592c74f87524a51e0766bf18b2d2813efb20790605f0b03 WatchSource:0}: Error finding container b4031663aebc38123592c74f87524a51e0766bf18b2d2813efb20790605f0b03: Status 404 returned error can't find the container with id b4031663aebc38123592c74f87524a51e0766bf18b2d2813efb20790605f0b03 Nov 02 22:14:29.275709 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:29.275689 5871 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:14:30.090547 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:30.090511 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" event={"ID":"a4bfaefa-f570-4bf9-abd6-45231654dafe","Type":"ContainerStarted","Data":"b4031663aebc38123592c74f87524a51e0766bf18b2d2813efb20790605f0b03"} Nov 02 22:14:32.101989 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:32.101890 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" event={"ID":"a4bfaefa-f570-4bf9-abd6-45231654dafe","Type":"ContainerStarted","Data":"231e8993e0242b8560d7600f577289bee5e27264dd57b775b37c91c1be8cfa0d"} Nov 02 22:14:32.213831 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:32.213775 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" podStartSLOduration=1.761055384 podStartE2EDuration="4.213757099s" totalImagesPullingTime="2.452701715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:32.213487914 +0000 UTC m=+495.025818093" watchObservedRunningTime="2026-11-02 22:14:32.213757099 +0000 UTC m=+495.026087282" Nov 02 22:14:32.895520 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:32.895477 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:32.926006 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:32.925974 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:35.267528 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:35.267481 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:36.151700 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.151662 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7"] Nov 02 22:14:36.155321 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.155297 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.250998 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.250967 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 02 22:14:36.253064 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.253040 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7"] Nov 02 22:14:36.284331 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.284279 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:36.324665 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.324629 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccce4075-668d-487a-8a75-fbd41fee023a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.324665 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.324666 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccce4075-668d-487a-8a75-fbd41fee023a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.324931 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.324754 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccce4075-668d-487a-8a75-fbd41fee023a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.324931 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.324843 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccce4075-668d-487a-8a75-fbd41fee023a-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.324931 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.324898 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbf84\" (UniqueName: \"kubernetes.io/projected/ccce4075-668d-487a-8a75-fbd41fee023a-kube-api-access-vbf84\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.324931 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.324917 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccce4075-668d-487a-8a75-fbd41fee023a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.425598 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.425504 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccce4075-668d-487a-8a75-fbd41fee023a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.425598 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.425546 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccce4075-668d-487a-8a75-fbd41fee023a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.425598 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.425584 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccce4075-668d-487a-8a75-fbd41fee023a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.425912 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.425619 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccce4075-668d-487a-8a75-fbd41fee023a-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.425912 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.425657 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbf84\" (UniqueName: \"kubernetes.io/projected/ccce4075-668d-487a-8a75-fbd41fee023a-kube-api-access-vbf84\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.425912 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.425682 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccce4075-668d-487a-8a75-fbd41fee023a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.426090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.426000 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccce4075-668d-487a-8a75-fbd41fee023a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.426150 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.426084 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccce4075-668d-487a-8a75-fbd41fee023a-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.426150 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.426137 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccce4075-668d-487a-8a75-fbd41fee023a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.427937 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.427912 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccce4075-668d-487a-8a75-fbd41fee023a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.428081 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.428065 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccce4075-668d-487a-8a75-fbd41fee023a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.472369 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.472331 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:14:36.483951 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.483921 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbf84\" (UniqueName: \"kubernetes.io/projected/ccce4075-668d-487a-8a75-fbd41fee023a-kube-api-access-vbf84\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7\" (UID: \"ccce4075-668d-487a-8a75-fbd41fee023a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:36.766817 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:36.766721 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:37.140661 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:37.140623 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7"] Nov 02 22:14:37.147839 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:14:37.147809 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccce4075_668d_487a_8a75_fbd41fee023a.slice/crio-c0f2ba4c30a7891447d76fcb3a9aa203c2b0365823648eaaffeee408aa6d684c WatchSource:0}: Error finding container c0f2ba4c30a7891447d76fcb3a9aa203c2b0365823648eaaffeee408aa6d684c: Status 404 returned error can't find the container with id c0f2ba4c30a7891447d76fcb3a9aa203c2b0365823648eaaffeee408aa6d684c Nov 02 22:14:38.135534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:38.135497 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" event={"ID":"ccce4075-668d-487a-8a75-fbd41fee023a","Type":"ContainerStarted","Data":"c0f2ba4c30a7891447d76fcb3a9aa203c2b0365823648eaaffeee408aa6d684c"} Nov 02 22:14:38.896467 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:38.896412 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:38.909103 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:38.909073 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:39.141862 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:39.141824 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" event={"ID":"ccce4075-668d-487a-8a75-fbd41fee023a","Type":"ContainerStarted","Data":"0513853febda092bd451ca37bb85442296e970412d87b828265c48077398469c"} Nov 02 22:14:39.153359 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:39.153326 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-8qtld" Nov 02 22:14:39.253806 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:39.253744 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podStartSLOduration=1.6532547549999999 podStartE2EDuration="3.253728332s" totalImagesPullingTime="1.600473577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:39.252007156 +0000 UTC m=+502.064337334" watchObservedRunningTime="2026-11-02 22:14:39.253728332 +0000 UTC m=+502.066058506" Nov 02 22:14:39.767023 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:39.766977 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:42.507315 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.507279 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl"] Nov 02 22:14:42.515790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.515757 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.566013 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.565976 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl"] Nov 02 22:14:42.583774 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.583741 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e8906d-5f46-4056-baa4-a1401866e0ad-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.583774 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.583777 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e8906d-5f46-4056-baa4-a1401866e0ad-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.584026 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.583810 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e8906d-5f46-4056-baa4-a1401866e0ad-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.584026 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.583854 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e8906d-5f46-4056-baa4-a1401866e0ad-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.584110 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.584029 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf82b\" (UniqueName: \"kubernetes.io/projected/57e8906d-5f46-4056-baa4-a1401866e0ad-kube-api-access-gf82b\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.584110 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.584086 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e8906d-5f46-4056-baa4-a1401866e0ad-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.609924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.609866 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 02 22:14:42.625443 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.625413 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:42.684815 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.684767 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e8906d-5f46-4056-baa4-a1401866e0ad-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.684815 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.684819 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e8906d-5f46-4056-baa4-a1401866e0ad-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.685103 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.684849 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e8906d-5f46-4056-baa4-a1401866e0ad-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.685103 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.684902 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e8906d-5f46-4056-baa4-a1401866e0ad-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.685103 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.684981 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf82b\" (UniqueName: \"kubernetes.io/projected/57e8906d-5f46-4056-baa4-a1401866e0ad-kube-api-access-gf82b\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.685103 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.685020 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e8906d-5f46-4056-baa4-a1401866e0ad-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.685299 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.685282 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e8906d-5f46-4056-baa4-a1401866e0ad-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.685400 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.685371 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e8906d-5f46-4056-baa4-a1401866e0ad-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.685486 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.685466 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e8906d-5f46-4056-baa4-a1401866e0ad-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.687187 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.687166 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e8906d-5f46-4056-baa4-a1401866e0ad-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.687440 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.687421 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e8906d-5f46-4056-baa4-a1401866e0ad-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.742835 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.742801 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf82b\" (UniqueName: \"kubernetes.io/projected/57e8906d-5f46-4056-baa4-a1401866e0ad-kube-api-access-gf82b\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tmtfl\" (UID: \"57e8906d-5f46-4056-baa4-a1401866e0ad\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:42.829207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:42.829108 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:43.203918 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:43.203864 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl"] Nov 02 22:14:43.210937 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:14:43.210908 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e8906d_5f46_4056_baa4_a1401866e0ad.slice/crio-a372c1427416f3ba0a96b6272b48d54bbdf36ffb0d9154b6e7b9dfecb2bb369c WatchSource:0}: Error finding container a372c1427416f3ba0a96b6272b48d54bbdf36ffb0d9154b6e7b9dfecb2bb369c: Status 404 returned error can't find the container with id a372c1427416f3ba0a96b6272b48d54bbdf36ffb0d9154b6e7b9dfecb2bb369c Nov 02 22:14:43.770265 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:14:43.770227 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:14:44.169913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:44.169869 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" event={"ID":"57e8906d-5f46-4056-baa4-a1401866e0ad","Type":"ContainerStarted","Data":"173b301de59fc16b27e6a878454e437683d470ea1945cb963e80fb3dd8359003"} Nov 02 22:14:44.170102 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:44.169933 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" event={"ID":"57e8906d-5f46-4056-baa4-a1401866e0ad","Type":"ContainerStarted","Data":"a372c1427416f3ba0a96b6272b48d54bbdf36ffb0d9154b6e7b9dfecb2bb369c"} Nov 02 22:14:44.281220 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:44.281152 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" podStartSLOduration=1.6758047619999998 podStartE2EDuration="2.281135729s" totalImagesPullingTime="605.330967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:44.279542082 +0000 UTC m=+507.091872260" watchObservedRunningTime="2026-11-02 22:14:44.281135729 +0000 UTC m=+507.093465908" Nov 02 22:14:44.830168 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:44.830111 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:44.832791 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:44.832762 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:14:45.704997 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:45.704962 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:46.767301 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:46.767265 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:46.779742 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:46.779715 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:47.195857 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:47.195826 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:14:49.453897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.453826 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m"] Nov 02 22:14:49.457932 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.457906 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.550380 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.550341 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5320d06b-d052-42e4-a0d1-be48e14ba7ee-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.550380 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.550385 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5320d06b-d052-42e4-a0d1-be48e14ba7ee-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.550615 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.550412 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5320d06b-d052-42e4-a0d1-be48e14ba7ee-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.550615 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.550538 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5320d06b-d052-42e4-a0d1-be48e14ba7ee-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.550682 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.550625 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5320d06b-d052-42e4-a0d1-be48e14ba7ee-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.550682 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.550658 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqk24\" (UniqueName: \"kubernetes.io/projected/5320d06b-d052-42e4-a0d1-be48e14ba7ee-kube-api-access-kqk24\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.552787 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.552759 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 02 22:14:49.554965 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.554938 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m"] Nov 02 22:14:49.594744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.594705 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:49.652031 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.651999 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5320d06b-d052-42e4-a0d1-be48e14ba7ee-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.652031 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.652036 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5320d06b-d052-42e4-a0d1-be48e14ba7ee-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.652245 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.652070 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5320d06b-d052-42e4-a0d1-be48e14ba7ee-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.652245 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.652105 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5320d06b-d052-42e4-a0d1-be48e14ba7ee-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.652245 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.652160 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5320d06b-d052-42e4-a0d1-be48e14ba7ee-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.652245 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.652195 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqk24\" (UniqueName: \"kubernetes.io/projected/5320d06b-d052-42e4-a0d1-be48e14ba7ee-kube-api-access-kqk24\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.652521 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.652494 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5320d06b-d052-42e4-a0d1-be48e14ba7ee-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.652687 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.652528 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5320d06b-d052-42e4-a0d1-be48e14ba7ee-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.652687 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.652535 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5320d06b-d052-42e4-a0d1-be48e14ba7ee-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.654411 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.654389 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5320d06b-d052-42e4-a0d1-be48e14ba7ee-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.654507 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.654449 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5320d06b-d052-42e4-a0d1-be48e14ba7ee-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.710313 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.710236 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqk24\" (UniqueName: \"kubernetes.io/projected/5320d06b-d052-42e4-a0d1-be48e14ba7ee-kube-api-access-kqk24\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-6sz2m\" (UID: \"5320d06b-d052-42e4-a0d1-be48e14ba7ee\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:49.770753 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:49.770705 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:50.153917 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:14:50.153846 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5320d06b_d052_42e4_a0d1_be48e14ba7ee.slice/crio-24187863cd0f8e4210c74de8c43c6846202968eeefb9463d596a684a692e446f WatchSource:0}: Error finding container 24187863cd0f8e4210c74de8c43c6846202968eeefb9463d596a684a692e446f: Status 404 returned error can't find the container with id 24187863cd0f8e4210c74de8c43c6846202968eeefb9463d596a684a692e446f Nov 02 22:14:50.164114 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:50.164083 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m"] Nov 02 22:14:50.202177 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:50.202143 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" event={"ID":"5320d06b-d052-42e4-a0d1-be48e14ba7ee","Type":"ContainerStarted","Data":"24187863cd0f8e4210c74de8c43c6846202968eeefb9463d596a684a692e446f"} Nov 02 22:14:51.208657 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:51.208614 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" event={"ID":"5320d06b-d052-42e4-a0d1-be48e14ba7ee","Type":"ContainerStarted","Data":"841712b3db0235eeeddf0481f29eaf28add8acb51f2f61d4feee85d88188f2a7"} Nov 02 22:14:51.320737 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:51.320676 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" podStartSLOduration=1.761731058 podStartE2EDuration="2.32065762s" totalImagesPullingTime="558.926562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:51.318236114 +0000 UTC m=+514.130566291" watchObservedRunningTime="2026-11-02 22:14:51.32065762 +0000 UTC m=+514.132987798" Nov 02 22:14:51.770918 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:51.770858 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:52.609729 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:52.609685 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:14:52.829280 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:52.829238 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:52.841939 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:52.841907 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:53.231261 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:53.231233 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" Nov 02 22:14:55.069365 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:55.069330 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:14:59.771129 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:59.771083 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:14:59.783830 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:14:59.783796 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:15:00.268214 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:00.268186 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-6sz2m" Nov 02 22:15:01.816549 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:01.816506 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:01.998616 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:01.998572 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:09.612551 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.612512 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d"] Nov 02 22:15:09.619839 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.619809 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.671823 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.671786 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 22:15:09.718088 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.718054 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-q825d\"" Nov 02 22:15:09.720318 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.720286 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d"] Nov 02 22:15:09.741723 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.741682 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrc4b\" (UniqueName: \"kubernetes.io/projected/8fd7034d-1b03-47bb-bd55-53bbd67ec740-kube-api-access-nrc4b\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.741723 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.741727 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd7034d-1b03-47bb-bd55-53bbd67ec740-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.741994 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.741837 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd7034d-1b03-47bb-bd55-53bbd67ec740-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.741994 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.741893 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd7034d-1b03-47bb-bd55-53bbd67ec740-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.741994 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.741978 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd7034d-1b03-47bb-bd55-53bbd67ec740-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.742111 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.742019 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd7034d-1b03-47bb-bd55-53bbd67ec740-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.742111 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.742046 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd7034d-1b03-47bb-bd55-53bbd67ec740-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.742111 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.742098 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd7034d-1b03-47bb-bd55-53bbd67ec740-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.742226 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.742116 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd7034d-1b03-47bb-bd55-53bbd67ec740-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.843651 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.843609 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrc4b\" (UniqueName: \"kubernetes.io/projected/8fd7034d-1b03-47bb-bd55-53bbd67ec740-kube-api-access-nrc4b\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.843858 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.843658 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd7034d-1b03-47bb-bd55-53bbd67ec740-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.843858 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.843719 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd7034d-1b03-47bb-bd55-53bbd67ec740-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.843858 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.843756 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd7034d-1b03-47bb-bd55-53bbd67ec740-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.843858 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.843794 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd7034d-1b03-47bb-bd55-53bbd67ec740-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.843858 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.843855 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd7034d-1b03-47bb-bd55-53bbd67ec740-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.844168 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.843915 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd7034d-1b03-47bb-bd55-53bbd67ec740-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.844168 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.843956 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd7034d-1b03-47bb-bd55-53bbd67ec740-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.844168 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.843984 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd7034d-1b03-47bb-bd55-53bbd67ec740-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.844317 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.844281 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd7034d-1b03-47bb-bd55-53bbd67ec740-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.844376 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.844328 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd7034d-1b03-47bb-bd55-53bbd67ec740-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.844445 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.844420 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd7034d-1b03-47bb-bd55-53bbd67ec740-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.844696 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.844673 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd7034d-1b03-47bb-bd55-53bbd67ec740-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.844696 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.844690 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd7034d-1b03-47bb-bd55-53bbd67ec740-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.846151 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.846129 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd7034d-1b03-47bb-bd55-53bbd67ec740-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.846442 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.846425 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd7034d-1b03-47bb-bd55-53bbd67ec740-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.901857 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.901774 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd7034d-1b03-47bb-bd55-53bbd67ec740-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.902019 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.901896 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrc4b\" (UniqueName: \"kubernetes.io/projected/8fd7034d-1b03-47bb-bd55-53bbd67ec740-kube-api-access-nrc4b\") pod \"maas-default-gateway-openshift-default-76f4876667-pf98d\" (UID: \"8fd7034d-1b03-47bb-bd55-53bbd67ec740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:09.933663 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:09.933624 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:10.362704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:10.362674 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d"] Nov 02 22:15:10.369195 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:15:10.369155 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd7034d_1b03_47bb_bd55_53bbd67ec740.slice/crio-a21d59826867f71c68b04cad407e03de2e1334169397aaaf7d44e52d238343ae WatchSource:0}: Error finding container a21d59826867f71c68b04cad407e03de2e1334169397aaaf7d44e52d238343ae: Status 404 returned error can't find the container with id a21d59826867f71c68b04cad407e03de2e1334169397aaaf7d44e52d238343ae Nov 02 22:15:11.315429 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:11.315388 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" event={"ID":"8fd7034d-1b03-47bb-bd55-53bbd67ec740","Type":"ContainerStarted","Data":"a21d59826867f71c68b04cad407e03de2e1334169397aaaf7d44e52d238343ae"} Nov 02 22:15:13.773918 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:15:13.773862 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:15:15.070140 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:15.070111 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:15:15.070405 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:15.070176 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:15:15.070405 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:15.070202 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:15:15.346977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:15.346858 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" event={"ID":"8fd7034d-1b03-47bb-bd55-53bbd67ec740","Type":"ContainerStarted","Data":"692127ac6fdb1b152da49f29dfd6e20da1ceafa05c5c23885f665bf23cdb4bcf"} Nov 02 22:15:15.514948 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:15.514855 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" podStartSLOduration=1.816854906 podStartE2EDuration="6.514831589s" totalImagesPullingTime="4.697976683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:15.507702088 +0000 UTC m=+538.320032257" watchObservedRunningTime="2026-11-02 22:15:15.514831589 +0000 UTC m=+538.327161772" Nov 02 22:15:15.934381 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:15.934339 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:16.935404 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:16.935370 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-pf98d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 22:15:16.935898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:16.935438 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" podUID="8fd7034d-1b03-47bb-bd55-53bbd67ec740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 22:15:17.934954 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:17.934915 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-pf98d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 22:15:17.935140 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:17.934974 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" podUID="8fd7034d-1b03-47bb-bd55-53bbd67ec740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 22:15:18.934053 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:18.934011 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-pf98d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 22:15:18.934451 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:18.934093 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" podUID="8fd7034d-1b03-47bb-bd55-53bbd67ec740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 22:15:19.630674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.630638 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-dhr65"] Nov 02 22:15:19.640995 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.640957 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" Nov 02 22:15:19.679632 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.679589 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:19.689811 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.689779 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-dhr65"] Nov 02 22:15:19.733563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.733518 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-wxg4f\"" Nov 02 22:15:19.733563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.733524 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 22:15:19.733778 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.733572 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 22:15:19.742118 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.742083 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fde9ac-d750-4e0f-829f-ecefe881bf9b-metrics-certs\") pod \"ai-gateway-controller-86c7698f44-dhr65\" (UID: \"d3fde9ac-d750-4e0f-829f-ecefe881bf9b\") " pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" Nov 02 22:15:19.742273 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.742135 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2w9j\" (UniqueName: \"kubernetes.io/projected/d3fde9ac-d750-4e0f-829f-ecefe881bf9b-kube-api-access-t2w9j\") pod \"ai-gateway-controller-86c7698f44-dhr65\" (UID: \"d3fde9ac-d750-4e0f-829f-ecefe881bf9b\") " pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" Nov 02 22:15:19.843172 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.843134 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fde9ac-d750-4e0f-829f-ecefe881bf9b-metrics-certs\") pod \"ai-gateway-controller-86c7698f44-dhr65\" (UID: \"d3fde9ac-d750-4e0f-829f-ecefe881bf9b\") " pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" Nov 02 22:15:19.843368 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.843186 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2w9j\" (UniqueName: \"kubernetes.io/projected/d3fde9ac-d750-4e0f-829f-ecefe881bf9b-kube-api-access-t2w9j\") pod \"ai-gateway-controller-86c7698f44-dhr65\" (UID: \"d3fde9ac-d750-4e0f-829f-ecefe881bf9b\") " pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" Nov 02 22:15:19.845729 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.845699 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fde9ac-d750-4e0f-829f-ecefe881bf9b-metrics-certs\") pod \"ai-gateway-controller-86c7698f44-dhr65\" (UID: \"d3fde9ac-d750-4e0f-829f-ecefe881bf9b\") " pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" Nov 02 22:15:19.877954 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.877473 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:19.901191 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.901107 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2w9j\" (UniqueName: \"kubernetes.io/projected/d3fde9ac-d750-4e0f-829f-ecefe881bf9b-kube-api-access-t2w9j\") pod \"ai-gateway-controller-86c7698f44-dhr65\" (UID: \"d3fde9ac-d750-4e0f-829f-ecefe881bf9b\") " pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" Nov 02 22:15:19.935075 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.935034 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-pf98d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 22:15:19.935477 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.935092 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" podUID="8fd7034d-1b03-47bb-bd55-53bbd67ec740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 22:15:19.979516 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:19.979457 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" Nov 02 22:15:20.370962 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:20.353071 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-dhr65"] Nov 02 22:15:20.381384 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:15:20.381349 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fde9ac_d750_4e0f_829f_ecefe881bf9b.slice/crio-507e79955aa021eaab348baab87e7438e1053da9bb86f055dd7c75ab3b08d23c WatchSource:0}: Error finding container 507e79955aa021eaab348baab87e7438e1053da9bb86f055dd7c75ab3b08d23c: Status 404 returned error can't find the container with id 507e79955aa021eaab348baab87e7438e1053da9bb86f055dd7c75ab3b08d23c Nov 02 22:15:20.401255 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:20.401216 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" event={"ID":"d3fde9ac-d750-4e0f-829f-ecefe881bf9b","Type":"ContainerStarted","Data":"507e79955aa021eaab348baab87e7438e1053da9bb86f055dd7c75ab3b08d23c"} Nov 02 22:15:20.923680 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:20.923648 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:20.934847 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:20.934817 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-pf98d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 22:15:20.934945 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:20.934902 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" podUID="8fd7034d-1b03-47bb-bd55-53bbd67ec740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 22:15:21.934793 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:21.934754 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-pf98d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 22:15:21.935294 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:21.934824 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" podUID="8fd7034d-1b03-47bb-bd55-53bbd67ec740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 22:15:23.785340 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:23.785304 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:23.786059 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:23.786029 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:23.787016 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:23.786994 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" Nov 02 22:15:24.726965 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:24.726868 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv"] Nov 02 22:15:24.730550 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:24.730528 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:24.828146 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:24.828110 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv"] Nov 02 22:15:24.915224 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:24.915184 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:24.915224 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:24.915223 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:24.915441 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:24.915324 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:24.915441 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:24.915389 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:24.915501 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:24.915486 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:24.915538 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:24.915517 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:24.915571 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:24.915539 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:24.915685 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:24.915671 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltd9w\" (UniqueName: \"kubernetes.io/projected/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-kube-api-access-ltd9w\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:24.915720 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:24.915705 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.016458 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.016362 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.016458 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.016408 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.016458 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.016425 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.016731 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.016485 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltd9w\" (UniqueName: \"kubernetes.io/projected/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-kube-api-access-ltd9w\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.016731 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.016506 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.016731 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.016529 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.016731 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.016554 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.016731 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.016597 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.016731 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.016662 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.017074 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.016937 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.017074 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.016972 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.017249 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.017225 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.017314 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.017253 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.017487 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.017455 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.018739 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.018718 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.019095 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.019069 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.074294 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.074253 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.074539 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.074510 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltd9w\" (UniqueName: \"kubernetes.io/projected/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-kube-api-access-ltd9w\") pod \"maas-default-gateway-openshift-default-76f4876667-9jnjv\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.343738 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.343648 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:25.777341 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.777302 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv"] Nov 02 22:15:25.783566 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:15:25.783530 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a69dbf_afb3_4e8d_ac96_2207095ca1c4.slice/crio-84c2e8b0217975fb4d3cb2ced9bfb04a1125b0b92976660ab06a45acecb34291 WatchSource:0}: Error finding container 84c2e8b0217975fb4d3cb2ced9bfb04a1125b0b92976660ab06a45acecb34291: Status 404 returned error can't find the container with id 84c2e8b0217975fb4d3cb2ced9bfb04a1125b0b92976660ab06a45acecb34291 Nov 02 22:15:25.785752 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.785718 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:15:25.785835 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.785793 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:15:25.785835 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:25.785826 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:15:26.442779 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:26.442739 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" event={"ID":"88a69dbf-afb3-4e8d-ac96-2207095ca1c4","Type":"ContainerStarted","Data":"82eeaf7a11fed848a1a90fe82a065af24fc28c312e194725423d26e5e7456793"} Nov 02 22:15:26.443229 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:26.442802 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" event={"ID":"88a69dbf-afb3-4e8d-ac96-2207095ca1c4","Type":"ContainerStarted","Data":"84c2e8b0217975fb4d3cb2ced9bfb04a1125b0b92976660ab06a45acecb34291"} Nov 02 22:15:26.603543 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:26.603481 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" podStartSLOduration=2.603468202 podStartE2EDuration="2.603468202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:26.601596989 +0000 UTC m=+549.413927178" watchObservedRunningTime="2026-11-02 22:15:26.603468202 +0000 UTC m=+549.415798380" Nov 02 22:15:27.344689 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:27.344652 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:27.346157 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:27.346131 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9jnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 22:15:27.346227 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:27.346195 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" podUID="88a69dbf-afb3-4e8d-ac96-2207095ca1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 22:15:28.344409 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:28.344321 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9jnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 22:15:28.344409 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:28.344390 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" podUID="88a69dbf-afb3-4e8d-ac96-2207095ca1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 22:15:28.481789 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:28.481749 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" event={"ID":"d3fde9ac-d750-4e0f-829f-ecefe881bf9b","Type":"ContainerStarted","Data":"cd989aa685c00325d86907d1fec057f22000f8760c9c4619f81194272001fcc5"} Nov 02 22:15:28.482037 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:28.481822 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" Nov 02 22:15:28.594755 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:28.594637 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" podStartSLOduration=1.983001694 podStartE2EDuration="9.594618262s" totalImagesPullingTime="7.611616568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:28.592783125 +0000 UTC m=+551.405113306" watchObservedRunningTime="2026-11-02 22:15:28.594618262 +0000 UTC m=+551.406948441" Nov 02 22:15:29.016507 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:29.016466 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:29.344713 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:29.344618 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9jnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 22:15:29.344713 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:29.344678 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" podUID="88a69dbf-afb3-4e8d-ac96-2207095ca1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 22:15:30.344606 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:30.344568 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9jnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 22:15:30.344798 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:30.344631 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" podUID="88a69dbf-afb3-4e8d-ac96-2207095ca1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 22:15:30.747865 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:30.747834 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:30.993801 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:30.993764 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:31.344925 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:31.344890 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9jnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 22:15:31.345359 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:31.344950 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" podUID="88a69dbf-afb3-4e8d-ac96-2207095ca1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 22:15:32.344084 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:32.344041 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-9jnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 02 22:15:32.344302 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:32.344118 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" podUID="88a69dbf-afb3-4e8d-ac96-2207095ca1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 02 22:15:34.138711 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:34.138672 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:34.139240 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:34.139214 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:34.140112 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:34.140091 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:15:34.862264 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:34.862198 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:35.615407 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:35.615370 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:39.488657 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:39.488627 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" Nov 02 22:15:39.844619 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:39.844518 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-54d8d5fc5c-6k894"] Nov 02 22:15:39.848499 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:39.848470 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-54d8d5fc5c-6k894" Nov 02 22:15:39.895370 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:39.895320 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqgjj\" (UniqueName: \"kubernetes.io/projected/313ca49b-f935-4457-a2a3-afd36e3477e5-kube-api-access-fqgjj\") pod \"ai-gateway-controller-54d8d5fc5c-6k894\" (UID: \"313ca49b-f935-4457-a2a3-afd36e3477e5\") " pod="opendatahub/ai-gateway-controller-54d8d5fc5c-6k894" Nov 02 22:15:39.895370 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:39.895372 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/313ca49b-f935-4457-a2a3-afd36e3477e5-metrics-certs\") pod \"ai-gateway-controller-54d8d5fc5c-6k894\" (UID: \"313ca49b-f935-4457-a2a3-afd36e3477e5\") " pod="opendatahub/ai-gateway-controller-54d8d5fc5c-6k894" Nov 02 22:15:39.903225 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:39.903182 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-54d8d5fc5c-6k894"] Nov 02 22:15:39.996541 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:39.996499 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqgjj\" (UniqueName: \"kubernetes.io/projected/313ca49b-f935-4457-a2a3-afd36e3477e5-kube-api-access-fqgjj\") pod \"ai-gateway-controller-54d8d5fc5c-6k894\" (UID: \"313ca49b-f935-4457-a2a3-afd36e3477e5\") " pod="opendatahub/ai-gateway-controller-54d8d5fc5c-6k894" Nov 02 22:15:39.996541 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:39.996542 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/313ca49b-f935-4457-a2a3-afd36e3477e5-metrics-certs\") pod \"ai-gateway-controller-54d8d5fc5c-6k894\" (UID: \"313ca49b-f935-4457-a2a3-afd36e3477e5\") " pod="opendatahub/ai-gateway-controller-54d8d5fc5c-6k894" Nov 02 22:15:39.999176 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:39.999142 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/313ca49b-f935-4457-a2a3-afd36e3477e5-metrics-certs\") pod \"ai-gateway-controller-54d8d5fc5c-6k894\" (UID: \"313ca49b-f935-4457-a2a3-afd36e3477e5\") " pod="opendatahub/ai-gateway-controller-54d8d5fc5c-6k894" Nov 02 22:15:40.054570 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:40.054532 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqgjj\" (UniqueName: \"kubernetes.io/projected/313ca49b-f935-4457-a2a3-afd36e3477e5-kube-api-access-fqgjj\") pod \"ai-gateway-controller-54d8d5fc5c-6k894\" (UID: \"313ca49b-f935-4457-a2a3-afd36e3477e5\") " pod="opendatahub/ai-gateway-controller-54d8d5fc5c-6k894" Nov 02 22:15:40.161627 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:40.161596 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-54d8d5fc5c-6k894" Nov 02 22:15:40.535274 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:40.535182 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-54d8d5fc5c-6k894"] Nov 02 22:15:40.541765 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:15:40.541735 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313ca49b_f935_4457_a2a3_afd36e3477e5.slice/crio-28a91223e5784bc255c8d80bef035e5c0f9a582c3d05e7252e53b82ee70305ca WatchSource:0}: Error finding container 28a91223e5784bc255c8d80bef035e5c0f9a582c3d05e7252e53b82ee70305ca: Status 404 returned error can't find the container with id 28a91223e5784bc255c8d80bef035e5c0f9a582c3d05e7252e53b82ee70305ca Nov 02 22:15:40.553729 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:40.553690 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-54d8d5fc5c-6k894" event={"ID":"313ca49b-f935-4457-a2a3-afd36e3477e5","Type":"ContainerStarted","Data":"28a91223e5784bc255c8d80bef035e5c0f9a582c3d05e7252e53b82ee70305ca"} Nov 02 22:15:41.568158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:41.568116 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-54d8d5fc5c-6k894" event={"ID":"313ca49b-f935-4457-a2a3-afd36e3477e5","Type":"ContainerStarted","Data":"e84a2dff8cae6c6e7538eb209e9db5e5e338a0620322601f9817c7adb32e6571"} Nov 02 22:15:41.568542 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:41.568417 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-54d8d5fc5c-6k894" Nov 02 22:15:41.677602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:41.677536 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-54d8d5fc5c-6k894" podStartSLOduration=1.768559732 podStartE2EDuration="2.677518773s" totalImagesPullingTime="908.959041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:41.676935493 +0000 UTC m=+564.489265671" watchObservedRunningTime="2026-11-02 22:15:41.677518773 +0000 UTC m=+564.489848954" Nov 02 22:15:43.771178 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:15:43.771139 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:15:46.846906 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:46.846854 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:52.576025 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:52.575989 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-54d8d5fc5c-6k894" Nov 02 22:15:52.716341 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:52.716290 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-dhr65"] Nov 02 22:15:52.716622 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:52.716569 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" podUID="d3fde9ac-d750-4e0f-829f-ecefe881bf9b" containerName="manager" containerID="cri-o://cd989aa685c00325d86907d1fec057f22000f8760c9c4619f81194272001fcc5" gracePeriod=30 Nov 02 22:15:52.970991 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:52.970961 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" Nov 02 22:15:53.129278 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:53.129246 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3fde9ac-d750-4e0f-829f-ecefe881bf9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fde9ac-d750-4e0f-829f-ecefe881bf9b-metrics-certs\") pod \"d3fde9ac-d750-4e0f-829f-ecefe881bf9b\" (UID: \"d3fde9ac-d750-4e0f-829f-ecefe881bf9b\") " Nov 02 22:15:53.129457 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:53.129355 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3fde9ac-d750-4e0f-829f-ecefe881bf9b-kube-api-access-t2w9j\" (UniqueName: \"kubernetes.io/projected/d3fde9ac-d750-4e0f-829f-ecefe881bf9b-kube-api-access-t2w9j\") pod \"d3fde9ac-d750-4e0f-829f-ecefe881bf9b\" (UID: \"d3fde9ac-d750-4e0f-829f-ecefe881bf9b\") " Nov 02 22:15:53.131484 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:53.131452 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3fde9ac-d750-4e0f-829f-ecefe881bf9b-metrics-certs" pod "d3fde9ac-d750-4e0f-829f-ecefe881bf9b" (UID: "d3fde9ac-d750-4e0f-829f-ecefe881bf9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:15:53.131578 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:53.131518 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fde9ac-d750-4e0f-829f-ecefe881bf9b-kube-api-access-t2w9j" pod "d3fde9ac-d750-4e0f-829f-ecefe881bf9b" (UID: "d3fde9ac-d750-4e0f-829f-ecefe881bf9b"). InnerVolumeSpecName "kube-api-access-t2w9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:53.231618 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:53.231585 5871 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fde9ac-d750-4e0f-829f-ecefe881bf9b-metrics-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:53.231618 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:53.231617 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2w9j\" (UniqueName: \"kubernetes.io/projected/d3fde9ac-d750-4e0f-829f-ecefe881bf9b-kube-api-access-t2w9j\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:53.636488 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:53.636453 5871 generic.go:356] "Generic (PLEG): container finished" podID="d3fde9ac-d750-4e0f-829f-ecefe881bf9b" containerID="cd989aa685c00325d86907d1fec057f22000f8760c9c4619f81194272001fcc5" exitCode=0 Nov 02 22:15:53.636919 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:53.636499 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" Nov 02 22:15:53.636919 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:53.636534 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" event={"ID":"d3fde9ac-d750-4e0f-829f-ecefe881bf9b","Type":"ContainerDied","Data":"cd989aa685c00325d86907d1fec057f22000f8760c9c4619f81194272001fcc5"} Nov 02 22:15:53.636919 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:53.636590 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-dhr65" event={"ID":"d3fde9ac-d750-4e0f-829f-ecefe881bf9b","Type":"ContainerDied","Data":"507e79955aa021eaab348baab87e7438e1053da9bb86f055dd7c75ab3b08d23c"} Nov 02 22:15:53.636919 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:53.636617 5871 scope.go:122] "RemoveContainer" containerID="cd989aa685c00325d86907d1fec057f22000f8760c9c4619f81194272001fcc5" Nov 02 22:15:53.646007 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:53.645987 5871 scope.go:122] "RemoveContainer" containerID="cd989aa685c00325d86907d1fec057f22000f8760c9c4619f81194272001fcc5" Nov 02 22:15:53.646304 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:15:53.646284 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd989aa685c00325d86907d1fec057f22000f8760c9c4619f81194272001fcc5\": container with ID starting with cd989aa685c00325d86907d1fec057f22000f8760c9c4619f81194272001fcc5 not found: ID does not exist" containerID="cd989aa685c00325d86907d1fec057f22000f8760c9c4619f81194272001fcc5" Nov 02 22:15:53.646363 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:53.646317 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd989aa685c00325d86907d1fec057f22000f8760c9c4619f81194272001fcc5"} err="failed to get container status \"cd989aa685c00325d86907d1fec057f22000f8760c9c4619f81194272001fcc5\": rpc error: code = NotFound desc = could not find container \"cd989aa685c00325d86907d1fec057f22000f8760c9c4619f81194272001fcc5\": container with ID starting with cd989aa685c00325d86907d1fec057f22000f8760c9c4619f81194272001fcc5 not found: ID does not exist" Nov 02 22:15:53.791370 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:53.791337 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-dhr65"] Nov 02 22:15:53.837406 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:53.837370 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-dhr65"] Nov 02 22:15:54.481009 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:54.480974 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:54.967157 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:54.967122 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x"] Nov 02 22:15:54.967603 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:54.967588 5871 cpu_manager.go:414] "containerMap: removing container" podUID="d3fde9ac-d750-4e0f-829f-ecefe881bf9b" containerName="manager" Nov 02 22:15:54.967643 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:54.967606 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fde9ac-d750-4e0f-829f-ecefe881bf9b" containerName="manager" Nov 02 22:15:54.967692 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:54.967683 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3fde9ac-d750-4e0f-829f-ecefe881bf9b" containerName="manager" Nov 02 22:15:54.972451 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:54.972432 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:54.977753 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:54.977725 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:55.048769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.048729 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.048992 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.048781 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.048992 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.048897 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.048992 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.048938 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h59fv\" (UniqueName: \"kubernetes.io/projected/4e6f0006-1702-4d03-89e5-8ed60a8203ee-kube-api-access-h59fv\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.049100 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.049013 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.049100 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.049069 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.049100 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.049088 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.049201 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.049134 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.049234 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.049202 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.071210 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.071167 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x"] Nov 02 22:15:55.150628 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.150581 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.150846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.150639 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h59fv\" (UniqueName: \"kubernetes.io/projected/4e6f0006-1702-4d03-89e5-8ed60a8203ee-kube-api-access-h59fv\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.150846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.150683 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.150846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.150742 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.150846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.150769 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.150846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.150799 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.150846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.150828 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.151164 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.150936 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.151164 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.150993 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.151326 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.151299 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.151527 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.151502 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.151680 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.151534 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.151784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.151637 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.152014 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.151993 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.153465 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.153439 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.153936 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.153914 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.209656 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.209623 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.209893 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.209854 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h59fv\" (UniqueName: \"kubernetes.io/projected/4e6f0006-1702-4d03-89e5-8ed60a8203ee-kube-api-access-h59fv\") pod \"maas-default-gateway-openshift-default-76f4876667-6vm9x\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.285664 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.285570 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:55.312382 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.312352 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3fde9ac-d750-4e0f-829f-ecefe881bf9b" path="/var/lib/kubelet/pods/d3fde9ac-d750-4e0f-829f-ecefe881bf9b/volumes" Nov 02 22:15:55.714679 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.714638 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x"] Nov 02 22:15:55.720933 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:15:55.720897 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6f0006_1702_4d03_89e5_8ed60a8203ee.slice/crio-6def666cd381cff55fe911cb8e0b3159bf27ec0e82b08cf310dba5a976a8a68a WatchSource:0}: Error finding container 6def666cd381cff55fe911cb8e0b3159bf27ec0e82b08cf310dba5a976a8a68a: Status 404 returned error can't find the container with id 6def666cd381cff55fe911cb8e0b3159bf27ec0e82b08cf310dba5a976a8a68a Nov 02 22:15:55.723187 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.723153 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:15:55.723265 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.723228 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:15:55.723265 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:55.723257 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:15:56.626605 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:56.626564 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:56.681147 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:56.680565 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" event={"ID":"4e6f0006-1702-4d03-89e5-8ed60a8203ee","Type":"ContainerStarted","Data":"852d7f3a7754e9c35abbe6b4a8fa82817400b83e0fee63e1e0101a064ada5d01"} Nov 02 22:15:56.681147 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:56.680698 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" event={"ID":"4e6f0006-1702-4d03-89e5-8ed60a8203ee","Type":"ContainerStarted","Data":"6def666cd381cff55fe911cb8e0b3159bf27ec0e82b08cf310dba5a976a8a68a"} Nov 02 22:15:56.810968 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:56.810928 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:56.844153 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:56.844098 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" podStartSLOduration=2.844082283 podStartE2EDuration="2.844082283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:56.841976181 +0000 UTC m=+579.654306355" watchObservedRunningTime="2026-11-02 22:15:56.844082283 +0000 UTC m=+579.656412543" Nov 02 22:15:57.285733 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:57.285700 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:15:57.287348 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:57.287314 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6vm9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 22:15:57.287475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:57.287375 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" podUID="4e6f0006-1702-4d03-89e5-8ed60a8203ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 22:15:58.286372 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:58.286336 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6vm9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 22:15:58.286801 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:58.286398 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" podUID="4e6f0006-1702-4d03-89e5-8ed60a8203ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 22:15:58.439925 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:58.439888 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:58.629214 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:58.629176 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:15:59.286914 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:59.286861 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6vm9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 22:15:59.287423 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:15:59.286957 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" podUID="4e6f0006-1702-4d03-89e5-8ed60a8203ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 22:16:00.286416 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:00.286368 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6vm9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 22:16:00.286738 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:00.286447 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" podUID="4e6f0006-1702-4d03-89e5-8ed60a8203ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 22:16:01.286734 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:01.286695 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6vm9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 22:16:01.286734 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:01.286796 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" podUID="4e6f0006-1702-4d03-89e5-8ed60a8203ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 22:16:02.286596 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:02.286552 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-6vm9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 22:16:02.286784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:02.286659 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" podUID="4e6f0006-1702-4d03-89e5-8ed60a8203ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 22:16:03.995923 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:03.995887 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:16:03.996802 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:03.996772 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:16:03.997556 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:03.997540 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:16:09.505120 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:09.505062 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:13.771004 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:16:13.770967 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:16:21.334074 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:21.334031 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:34.899301 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:34.899253 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:35.077373 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.077318 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:35.460814 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.460773 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5"] Nov 02 22:16:35.464954 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.464926 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.516258 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.516207 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-49a557-openshift-default-dockercfg-lgv8v\"" Nov 02 22:16:35.564565 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.564533 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5"] Nov 02 22:16:35.579704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.574133 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.579704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.578617 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.579704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.578694 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.579704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.578749 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.579704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.578784 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.579704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.578828 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.579704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.578853 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjsnw\" (UniqueName: \"kubernetes.io/projected/291b0b67-ee70-4d16-ac08-8064e7f0fef1-kube-api-access-cjsnw\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.579704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.578925 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.579704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.578983 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.591456 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.591425 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:35.679961 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.679920 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.680156 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.680078 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.680156 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.680124 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.680281 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.680172 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.680281 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.680200 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.680281 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.680241 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.680281 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.680268 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjsnw\" (UniqueName: \"kubernetes.io/projected/291b0b67-ee70-4d16-ac08-8064e7f0fef1-kube-api-access-cjsnw\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.680475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.680303 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.680475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.680353 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.680475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.680388 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.680699 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.680673 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.680772 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.680729 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.680938 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.680916 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.681183 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.681159 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.682504 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.682483 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.682725 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.682703 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.740004 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.739926 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.740167 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.740069 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjsnw\" (UniqueName: \"kubernetes.io/projected/291b0b67-ee70-4d16-ac08-8064e7f0fef1-kube-api-access-cjsnw\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.779935 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.779858 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:35.842962 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:35.842926 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:36.028636 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.028503 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:36.106701 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.106658 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7"] Nov 02 22:16:36.110832 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.110805 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.160261 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.160216 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8263a4-openshift-default-dockercfg-t2p57\"" Nov 02 22:16:36.170362 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.170326 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7"] Nov 02 22:16:36.206427 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.206390 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:36.219709 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.219673 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5"] Nov 02 22:16:36.227181 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:16:36.227152 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291b0b67_ee70_4d16_ac08_8064e7f0fef1.slice/crio-42f78971ec3be67b6feb4f5061b19e3e733c5593dbf6b0d7a2ccc7edf3d03637 WatchSource:0}: Error finding container 42f78971ec3be67b6feb4f5061b19e3e733c5593dbf6b0d7a2ccc7edf3d03637: Status 404 returned error can't find the container with id 42f78971ec3be67b6feb4f5061b19e3e733c5593dbf6b0d7a2ccc7edf3d03637 Nov 02 22:16:36.229508 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.229472 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:16:36.229601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.229544 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:16:36.229601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.229577 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:16:36.287463 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.287423 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae99bb6-0aee-4ef6-a938-f189add936ed-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.287630 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.287483 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.287630 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.287596 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.287723 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.287673 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.287723 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.287698 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx6h9\" (UniqueName: \"kubernetes.io/projected/fae99bb6-0aee-4ef6-a938-f189add936ed-kube-api-access-jx6h9\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.287786 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.287773 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.287828 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.287791 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.287944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.287925 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.288009 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.287953 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.386582 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.386539 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:36.388609 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.388578 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.388609 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.388609 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.388825 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.388668 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae99bb6-0aee-4ef6-a938-f189add936ed-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.388825 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.388690 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.388825 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.388724 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.389041 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.388815 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.389041 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.388861 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx6h9\" (UniqueName: \"kubernetes.io/projected/fae99bb6-0aee-4ef6-a938-f189add936ed-kube-api-access-jx6h9\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.389041 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.388941 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.389041 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.388972 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.389284 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.389113 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.389390 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.389360 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.389492 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.389470 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae99bb6-0aee-4ef6-a938-f189add936ed-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.389604 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.389585 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.389755 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.389731 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.391427 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.391395 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.391554 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.391474 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.447517 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.447482 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.447672 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.447571 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx6h9\" (UniqueName: \"kubernetes.io/projected/fae99bb6-0aee-4ef6-a938-f189add936ed-kube-api-access-jx6h9\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.562711 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.562635 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:36.724751 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.724712 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:36.750630 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.750600 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:36.758847 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.758815 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh"] Nov 02 22:16:36.762798 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.762774 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.813805 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.813696 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3224ea-openshift-default-dockercfg-2whsf\"" Nov 02 22:16:36.822085 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.822049 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh"] Nov 02 22:16:36.896117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.896081 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.896288 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.896144 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.896288 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.896250 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.896369 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.896296 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.896403 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.896363 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.896403 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.896398 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.896469 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.896425 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.896503 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.896474 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4839ef24-050d-41ea-95c3-7713d6df39a0-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.896567 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.896554 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct5hl\" (UniqueName: \"kubernetes.io/projected/4839ef24-050d-41ea-95c3-7713d6df39a0-kube-api-access-ct5hl\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.911275 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.911239 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" event={"ID":"291b0b67-ee70-4d16-ac08-8064e7f0fef1","Type":"ContainerStarted","Data":"6057643f29e05714c7f8cc154f4354fd969ce2d0694bc3aa73440ea0147c3764"} Nov 02 22:16:36.911503 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.911292 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" event={"ID":"291b0b67-ee70-4d16-ac08-8064e7f0fef1","Type":"ContainerStarted","Data":"42f78971ec3be67b6feb4f5061b19e3e733c5593dbf6b0d7a2ccc7edf3d03637"} Nov 02 22:16:36.939609 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.939571 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:36.997804 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.997766 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.997804 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.997806 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.998095 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.997829 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.998095 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.997852 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.998095 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.997980 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4839ef24-050d-41ea-95c3-7713d6df39a0-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.998095 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.998025 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct5hl\" (UniqueName: \"kubernetes.io/projected/4839ef24-050d-41ea-95c3-7713d6df39a0-kube-api-access-ct5hl\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.998095 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.998080 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.998354 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.998155 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.998354 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.998219 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.998354 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.998261 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.998354 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.998284 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.998354 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.998333 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.998609 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.998557 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:36.998704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:36.998676 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4839ef24-050d-41ea-95c3-7713d6df39a0-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:37.000428 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.000402 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:37.000526 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.000505 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:37.027727 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.027653 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" podStartSLOduration=2.027634039 podStartE2EDuration="2.027634039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:37.025297104 +0000 UTC m=+619.837627308" watchObservedRunningTime="2026-11-02 22:16:37.027634039 +0000 UTC m=+619.839964290" Nov 02 22:16:37.057805 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.057769 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct5hl\" (UniqueName: \"kubernetes.io/projected/4839ef24-050d-41ea-95c3-7713d6df39a0-kube-api-access-ct5hl\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:37.058292 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.058190 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:37.079131 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.079049 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:37.161716 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.161679 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7"] Nov 02 22:16:37.167611 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:16:37.167582 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae99bb6_0aee_4ef6_a938_f189add936ed.slice/crio-07cc786c9170f3a8f7cca19cfdec23f366b68f86a980b6962a06539c9071b16e WatchSource:0}: Error finding container 07cc786c9170f3a8f7cca19cfdec23f366b68f86a980b6962a06539c9071b16e: Status 404 returned error can't find the container with id 07cc786c9170f3a8f7cca19cfdec23f366b68f86a980b6962a06539c9071b16e Nov 02 22:16:37.170415 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.170372 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:16:37.170535 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.170465 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:16:37.170535 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.170507 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:16:37.173092 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.173067 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:37.349861 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.349775 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:37.468941 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.468901 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh"] Nov 02 22:16:37.481847 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.481808 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:16:37.481983 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.481915 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:16:37.481983 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.481962 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:16:37.529065 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.529035 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:37.715542 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.715503 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:37.780396 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.780361 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:37.792193 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.792163 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:37.911509 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.911472 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:37.917926 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.917859 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" event={"ID":"4839ef24-050d-41ea-95c3-7713d6df39a0","Type":"ContainerStarted","Data":"615c7c1e55c9aaf2754a2fcef8f78957d7abab0d09e57361dc6ca9e916b58d2c"} Nov 02 22:16:37.918114 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.917956 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" event={"ID":"4839ef24-050d-41ea-95c3-7713d6df39a0","Type":"ContainerStarted","Data":"5ba4ae88cdcb50cf8c94528726a7493674bf506f0aaac726f89fdd222d34b4f1"} Nov 02 22:16:37.919607 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.919577 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" event={"ID":"fae99bb6-0aee-4ef6-a938-f189add936ed","Type":"ContainerStarted","Data":"aa30db04432008adadfc0c297ab3f6fce700209c999d2fec1716a46364a4e4b9"} Nov 02 22:16:37.919721 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.919629 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" event={"ID":"fae99bb6-0aee-4ef6-a938-f189add936ed","Type":"ContainerStarted","Data":"07cc786c9170f3a8f7cca19cfdec23f366b68f86a980b6962a06539c9071b16e"} Nov 02 22:16:37.919906 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.919862 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:37.920913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:37.920868 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:16:38.036282 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:38.036168 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" podStartSLOduration=2.036152377 podStartE2EDuration="2.036152377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:38.034084502 +0000 UTC m=+620.846414680" watchObservedRunningTime="2026-11-02 22:16:38.036152377 +0000 UTC m=+620.848482632" Nov 02 22:16:38.080286 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:38.080245 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:38.085702 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:38.085673 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:38.267094 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:38.266985 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" podStartSLOduration=2.266966212 podStartE2EDuration="2.266966212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:38.263095396 +0000 UTC m=+621.075425577" watchObservedRunningTime="2026-11-02 22:16:38.266966212 +0000 UTC m=+621.079296392" Nov 02 22:16:38.725075 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:38.725035 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:38.730219 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:38.730184 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:38.925480 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:38.925446 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:38.925670 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:38.925506 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:38.926178 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:38.926157 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:16:38.926391 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:38.926374 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:16:39.917602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:39.917569 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:40.100521 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:40.100486 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:40.287498 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:40.287404 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:40.786687 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:40.786655 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:40.981121 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:40.981089 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:41.166123 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:41.166081 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:41.757535 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:41.757468 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:41.935038 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:41.935001 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:42.118914 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:42.118856 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:42.353827 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:42.353792 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:42.536899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:42.536804 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:42.723008 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:42.722958 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:43.770288 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:16:43.770248 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:16:44.082794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:44.082705 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:44.303090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:44.303055 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:44.484157 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:44.484123 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:45.766003 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:45.765943 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:45.953841 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:45.953803 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:46.143944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:46.143906 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:46.429432 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:46.429342 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:46.608219 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:46.608187 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:46.790206 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:46.790118 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:51.535549 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.535508 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7"] Nov 02 22:16:51.539500 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.539482 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:16:51.593340 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.593305 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-fe810f-serving-cert\"" Nov 02 22:16:51.639421 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.639390 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7"] Nov 02 22:16:51.645592 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.645572 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-metrics-certs\") pod \"maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:16:51.645663 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.645614 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03723b8-b2e9-4f09-9054-c069f1f5f45c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:16:51.645754 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.645741 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:16:51.645800 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.645790 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwgwc\" (UniqueName: \"kubernetes.io/projected/b03723b8-b2e9-4f09-9054-c069f1f5f45c-kube-api-access-dwgwc\") pod \"maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:16:51.746911 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.746849 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:16:51.747089 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.746937 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwgwc\" (UniqueName: \"kubernetes.io/projected/b03723b8-b2e9-4f09-9054-c069f1f5f45c-kube-api-access-dwgwc\") pod \"maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:16:51.747089 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.746985 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-metrics-certs\") pod \"maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:16:51.747089 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.747014 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03723b8-b2e9-4f09-9054-c069f1f5f45c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:16:51.747089 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:16:51.747022 5871 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-serving-cert: secret "maas-api-e2e-worker-w5-fe810f-serving-cert" not found Nov 02 22:16:51.747255 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:16:51.747133 5871 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-maas-api-tls podName:b03723b8-b2e9-4f09-9054-c069f1f5f45c nodeName:}" failed. No retries permitted until 2026-11-02 22:16:52.247110811 +0000 UTC m=+635.059440968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-maas-api-tls") pod "maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" (UID: "b03723b8-b2e9-4f09-9054-c069f1f5f45c") : secret "maas-api-e2e-worker-w5-fe810f-serving-cert" not found Nov 02 22:16:51.747634 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.747618 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03723b8-b2e9-4f09-9054-c069f1f5f45c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:16:51.749563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.749546 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-metrics-certs\") pod \"maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:16:51.760365 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.760341 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:51.807324 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.807254 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwgwc\" (UniqueName: \"kubernetes.io/projected/b03723b8-b2e9-4f09-9054-c069f1f5f45c-kube-api-access-dwgwc\") pod \"maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:16:51.940010 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:51.939959 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:52.123318 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:52.123281 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:52.253388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:52.253352 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:16:52.255803 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:52.255777 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:16:52.452807 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:52.452730 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:16:52.493304 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:52.493268 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:52.665399 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:52.665369 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:52.827567 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:52.827482 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7"] Nov 02 22:16:53.012640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:53.012603 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" event={"ID":"b03723b8-b2e9-4f09-9054-c069f1f5f45c","Type":"ContainerStarted","Data":"791c11ba52cca852dd74ee11ed9582ac07e48b0bccf2184dac59ec411d412375"} Nov 02 22:16:54.019770 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:54.019732 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" event={"ID":"b03723b8-b2e9-4f09-9054-c069f1f5f45c","Type":"ContainerStarted","Data":"3502d501c38037f70e8dfda6944e3aad15054be13451225808c698871a7b146b"} Nov 02 22:16:54.020237 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:54.019916 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:16:54.131475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:54.131419 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" podStartSLOduration=2.19627552 podStartE2EDuration="3.13140452s" totalImagesPullingTime="935.129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:54.130799464 +0000 UTC m=+636.943129642" watchObservedRunningTime="2026-11-02 22:16:54.13140452 +0000 UTC m=+636.943734697" Nov 02 22:16:54.356333 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:54.356296 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:16:56.982178 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:56.982133 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c"] Nov 02 22:16:56.985846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:56.985824 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:16:57.007372 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.007332 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhspx\" (UniqueName: \"kubernetes.io/projected/23563615-f0f4-4cb4-9159-815000b8fa24-kube-api-access-lhspx\") pod \"maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:16:57.007547 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.007462 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-metrics-certs\") pod \"maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:16:57.007547 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.007515 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:16:57.007668 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.007593 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23563615-f0f4-4cb4-9159-815000b8fa24-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:16:57.034658 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.034617 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-3224ea-serving-cert\"" Nov 02 22:16:57.083287 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.083246 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c"] Nov 02 22:16:57.108454 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.108415 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhspx\" (UniqueName: \"kubernetes.io/projected/23563615-f0f4-4cb4-9159-815000b8fa24-kube-api-access-lhspx\") pod \"maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:16:57.108639 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.108489 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-metrics-certs\") pod \"maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:16:57.108639 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.108532 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:16:57.108639 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.108567 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23563615-f0f4-4cb4-9159-815000b8fa24-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:16:57.108758 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:16:57.108668 5871 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-serving-cert: secret "maas-api-e2e-worker-w4-3224ea-serving-cert" not found Nov 02 22:16:57.108758 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:16:57.108732 5871 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-maas-api-tls podName:23563615-f0f4-4cb4-9159-815000b8fa24 nodeName:}" failed. No retries permitted until 2026-11-02 22:16:57.608716885 +0000 UTC m=+640.421047042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-maas-api-tls") pod "maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" (UID: "23563615-f0f4-4cb4-9159-815000b8fa24") : secret "maas-api-e2e-worker-w4-3224ea-serving-cert" not found Nov 02 22:16:57.109362 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.109338 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23563615-f0f4-4cb4-9159-815000b8fa24-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:16:57.111133 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.111112 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-metrics-certs\") pod \"maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:16:57.180712 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.180675 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhspx\" (UniqueName: \"kubernetes.io/projected/23563615-f0f4-4cb4-9159-815000b8fa24-kube-api-access-lhspx\") pod \"maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:16:57.232267 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.232181 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:57.422578 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.422545 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:57.609233 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.609148 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:57.613506 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.613479 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:16:57.615975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.615955 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:16:57.892138 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.892103 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:57.899144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:57.899114 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:16:58.078630 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:58.078451 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:16:58.292203 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:58.292118 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c"] Nov 02 22:16:58.299292 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:16:58.299263 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23563615_f0f4_4cb4_9159_815000b8fa24.slice/crio-9fcc6b298849e30dd8e9c8abb16d7dc9a17473ae8f07e78fc34aaf23a2e9e0b4 WatchSource:0}: Error finding container 9fcc6b298849e30dd8e9c8abb16d7dc9a17473ae8f07e78fc34aaf23a2e9e0b4: Status 404 returned error can't find the container with id 9fcc6b298849e30dd8e9c8abb16d7dc9a17473ae8f07e78fc34aaf23a2e9e0b4 Nov 02 22:16:59.085764 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:16:59.085709 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" event={"ID":"23563615-f0f4-4cb4-9159-815000b8fa24","Type":"ContainerStarted","Data":"9fcc6b298849e30dd8e9c8abb16d7dc9a17473ae8f07e78fc34aaf23a2e9e0b4"} Nov 02 22:17:00.030974 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:00.030943 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:17:00.105960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:00.097229 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" event={"ID":"23563615-f0f4-4cb4-9159-815000b8fa24","Type":"ContainerStarted","Data":"19308e84d0e0bb0ba8b3e6b45e6bf5839356f8bc3cab948d8f673d5d0066a3f8"} Nov 02 22:17:00.105960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:00.097555 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:17:00.257205 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:00.257148 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" podStartSLOduration=3.350652975 podStartE2EDuration="4.25712444s" totalImagesPullingTime="906.471465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:00.254695808 +0000 UTC m=+643.067025986" watchObservedRunningTime="2026-11-02 22:17:00.25712444 +0000 UTC m=+643.069454634" Nov 02 22:17:02.187811 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:02.187778 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:02.802980 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:02.802943 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:02.960324 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:02.960284 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:17:02.982083 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:02.982024 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:04.127993 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:04.127956 5871 generic.go:356] "Generic (PLEG): container finished" podID="291b0b67-ee70-4d16-ac08-8064e7f0fef1" containerID="6057643f29e05714c7f8cc154f4354fd969ce2d0694bc3aa73440ea0147c3764" exitCode=0 Nov 02 22:17:04.128449 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:04.128030 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" event={"ID":"291b0b67-ee70-4d16-ac08-8064e7f0fef1","Type":"ContainerDied","Data":"6057643f29e05714c7f8cc154f4354fd969ce2d0694bc3aa73440ea0147c3764"} Nov 02 22:17:04.128449 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:04.128432 5871 scope.go:122] "RemoveContainer" containerID="6057643f29e05714c7f8cc154f4354fd969ce2d0694bc3aa73440ea0147c3764" Nov 02 22:17:04.129847 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:04.129817 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:04.129983 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:04.129905 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:04.129983 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:04.129945 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:05.136042 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:05.136006 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" event={"ID":"291b0b67-ee70-4d16-ac08-8064e7f0fef1","Type":"ContainerStarted","Data":"f452c03041a06a856d521de95c28b341058913d28d9663f008cc7f3f5ccb23ca"} Nov 02 22:17:05.784898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:05.784521 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:17:05.784898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:05.784673 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 22:17:05.784898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:05.784720 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" podUID="291b0b67-ee70-4d16-ac08-8064e7f0fef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 22:17:06.111865 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:06.111781 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:17:06.782928 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:06.782361 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 22:17:06.782928 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:06.782432 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" podUID="291b0b67-ee70-4d16-ac08-8064e7f0fef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 22:17:07.215520 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:07.215359 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:07.404336 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:07.404296 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:07.591013 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:07.590842 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:07.784463 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:07.784426 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 22:17:07.785124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:07.785079 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" podUID="291b0b67-ee70-4d16-ac08-8064e7f0fef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 22:17:07.920174 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:07.920144 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:17:08.780413 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:08.780365 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 22:17:08.780724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:08.780687 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" podUID="291b0b67-ee70-4d16-ac08-8064e7f0fef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 22:17:09.781298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:09.781252 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 22:17:09.781694 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:09.781326 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" podUID="291b0b67-ee70-4d16-ac08-8064e7f0fef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 22:17:10.786354 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:10.786320 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:17:10.787554 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:10.787534 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:17:12.116788 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:12.116753 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:12.299842 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:12.299808 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:12.478274 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:12.478179 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:12.757101 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:12.756979 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:12.937956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:12.937909 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:13.217440 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:13.217407 5871 generic.go:356] "Generic (PLEG): container finished" podID="4839ef24-050d-41ea-95c3-7713d6df39a0" containerID="615c7c1e55c9aaf2754a2fcef8f78957d7abab0d09e57361dc6ca9e916b58d2c" exitCode=0 Nov 02 22:17:13.217845 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:13.217475 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" event={"ID":"4839ef24-050d-41ea-95c3-7713d6df39a0","Type":"ContainerDied","Data":"615c7c1e55c9aaf2754a2fcef8f78957d7abab0d09e57361dc6ca9e916b58d2c"} Nov 02 22:17:13.217911 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:13.217866 5871 scope.go:122] "RemoveContainer" containerID="615c7c1e55c9aaf2754a2fcef8f78957d7abab0d09e57361dc6ca9e916b58d2c" Nov 02 22:17:13.219226 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:13.219198 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:13.219339 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:13.219285 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:13.219408 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:13.219338 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:13.771306 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:17:13.771266 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:17:14.225329 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:14.225291 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" event={"ID":"4839ef24-050d-41ea-95c3-7713d6df39a0","Type":"ContainerStarted","Data":"b908cb8162fd2b56349ad3a8178033c2aee7c22ad28c7108888f647e68bd2b08"} Nov 02 22:17:15.086895 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:15.086278 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:17:15.086895 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:15.086410 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 22:17:15.086895 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:15.086468 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" podUID="4839ef24-050d-41ea-95c3-7713d6df39a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 22:17:16.079446 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:16.079405 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 22:17:16.079911 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:16.079467 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" podUID="4839ef24-050d-41ea-95c3-7713d6df39a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 22:17:17.079677 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:17.079636 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 22:17:17.080150 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:17.079699 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" podUID="4839ef24-050d-41ea-95c3-7713d6df39a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 22:17:17.323643 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:17.323609 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:17.844227 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:17.844186 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:18.021098 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:18.021062 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:18.080054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:18.080018 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 22:17:18.080451 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:18.080083 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" podUID="4839ef24-050d-41ea-95c3-7713d6df39a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 22:17:18.220781 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:18.220746 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47"] Nov 02 22:17:18.223779 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:18.223761 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" Nov 02 22:17:18.276974 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:18.276940 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-8263a4\"" Nov 02 22:17:18.283897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:18.281996 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47"] Nov 02 22:17:18.322408 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:18.322379 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-8263a4-dockercfg-dp55f\"" Nov 02 22:17:18.414837 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:18.414799 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk75v\" (UniqueName: \"kubernetes.io/projected/e449a219-b519-48b7-a4b0-b9dc589cdeb3-kube-api-access-pk75v\") pod \"payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47\" (UID: \"e449a219-b519-48b7-a4b0-b9dc589cdeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" Nov 02 22:17:18.415043 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:18.415020 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e449a219-b519-48b7-a4b0-b9dc589cdeb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47\" (UID: \"e449a219-b519-48b7-a4b0-b9dc589cdeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" Nov 02 22:17:18.516643 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:18.516550 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e449a219-b519-48b7-a4b0-b9dc589cdeb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47\" (UID: \"e449a219-b519-48b7-a4b0-b9dc589cdeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" Nov 02 22:17:18.516643 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:18.516637 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk75v\" (UniqueName: \"kubernetes.io/projected/e449a219-b519-48b7-a4b0-b9dc589cdeb3-kube-api-access-pk75v\") pod \"payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47\" (UID: \"e449a219-b519-48b7-a4b0-b9dc589cdeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" Nov 02 22:17:18.517541 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:18.517514 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e449a219-b519-48b7-a4b0-b9dc589cdeb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47\" (UID: \"e449a219-b519-48b7-a4b0-b9dc589cdeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" Nov 02 22:17:18.579155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:18.579118 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk75v\" (UniqueName: \"kubernetes.io/projected/e449a219-b519-48b7-a4b0-b9dc589cdeb3-kube-api-access-pk75v\") pod \"payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47\" (UID: \"e449a219-b519-48b7-a4b0-b9dc589cdeb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" Nov 02 22:17:18.836694 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:18.836603 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" Nov 02 22:17:19.080431 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:19.080393 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 22:17:19.080938 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:19.080454 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" podUID="4839ef24-050d-41ea-95c3-7713d6df39a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 22:17:19.240976 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:19.240933 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47"] Nov 02 22:17:19.247071 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:17:19.247038 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode449a219_b519_48b7_a4b0_b9dc589cdeb3.slice/crio-6c94f1945da0d311c9a3695deb92ecc2febeed707bc5da2c9ce0e3f2586e8a2b WatchSource:0}: Error finding container 6c94f1945da0d311c9a3695deb92ecc2febeed707bc5da2c9ce0e3f2586e8a2b: Status 404 returned error can't find the container with id 6c94f1945da0d311c9a3695deb92ecc2febeed707bc5da2c9ce0e3f2586e8a2b Nov 02 22:17:19.260916 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:19.260816 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" event={"ID":"e449a219-b519-48b7-a4b0-b9dc589cdeb3","Type":"ContainerStarted","Data":"6c94f1945da0d311c9a3695deb92ecc2febeed707bc5da2c9ce0e3f2586e8a2b"} Nov 02 22:17:20.084473 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:20.084438 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:17:20.084899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:20.084732 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:17:20.085411 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:20.085392 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:17:21.554564 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:21.554527 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:17:21.614625 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:21.614584 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:21.900161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:21.900125 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4"] Nov 02 22:17:21.907528 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:21.907493 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:21.950593 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:21.950557 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p2g5\" (UniqueName: \"kubernetes.io/projected/16b4519b-9df1-4c63-9f22-d46e804efef4-kube-api-access-4p2g5\") pod \"maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:21.950785 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:21.950622 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b4519b-9df1-4c63-9f22-d46e804efef4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:21.950785 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:21.950729 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-metrics-certs\") pod \"maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:21.950862 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:21.950807 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:21.956525 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:21.956493 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-98a222-serving-cert\"" Nov 02 22:17:22.005355 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:22.005313 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4"] Nov 02 22:17:22.051603 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:22.051565 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p2g5\" (UniqueName: \"kubernetes.io/projected/16b4519b-9df1-4c63-9f22-d46e804efef4-kube-api-access-4p2g5\") pod \"maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:22.051790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:22.051631 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b4519b-9df1-4c63-9f22-d46e804efef4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:22.051790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:22.051667 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-metrics-certs\") pod \"maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:22.051790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:22.051694 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:22.052001 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:17:22.051812 5871 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-serving-cert: secret "maas-api-e2e-worker-w3-98a222-serving-cert" not found Nov 02 22:17:22.052001 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:17:22.051895 5871 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-maas-api-tls podName:16b4519b-9df1-4c63-9f22-d46e804efef4 nodeName:}" failed. No retries permitted until 2026-11-02 22:17:22.55186233 +0000 UTC m=+665.364192493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-maas-api-tls") pod "maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" (UID: "16b4519b-9df1-4c63-9f22-d46e804efef4") : secret "maas-api-e2e-worker-w3-98a222-serving-cert" not found Nov 02 22:17:22.052531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:22.052506 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b4519b-9df1-4c63-9f22-d46e804efef4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:22.054580 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:22.054544 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-metrics-certs\") pod \"maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:22.110440 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:22.110402 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p2g5\" (UniqueName: \"kubernetes.io/projected/16b4519b-9df1-4c63-9f22-d46e804efef4-kube-api-access-4p2g5\") pod \"maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:22.129265 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:22.129231 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:22.330586 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:22.330501 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:22.578525 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:22.578482 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:22.580073 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:22.580036 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:22.581795 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:22.581738 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:22.822753 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:22.822717 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:23.233314 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:23.233277 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4"] Nov 02 22:17:23.240291 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:17:23.240255 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b4519b_9df1_4c63_9f22_d46e804efef4.slice/crio-2ce6b2e9a3dcc8953df976c09fbe8d6d3b2c19986f65b4b61b7cb63ff5669355 WatchSource:0}: Error finding container 2ce6b2e9a3dcc8953df976c09fbe8d6d3b2c19986f65b4b61b7cb63ff5669355: Status 404 returned error can't find the container with id 2ce6b2e9a3dcc8953df976c09fbe8d6d3b2c19986f65b4b61b7cb63ff5669355 Nov 02 22:17:23.287295 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:23.287250 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" event={"ID":"e449a219-b519-48b7-a4b0-b9dc589cdeb3","Type":"ContainerStarted","Data":"856366419bb8331212bf34e7fd673fb0f2108883d28eda984a35687f02bdcf72"} Nov 02 22:17:23.287497 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:23.287320 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" Nov 02 22:17:23.288597 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:23.288573 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" event={"ID":"16b4519b-9df1-4c63-9f22-d46e804efef4","Type":"ContainerStarted","Data":"2ce6b2e9a3dcc8953df976c09fbe8d6d3b2c19986f65b4b61b7cb63ff5669355"} Nov 02 22:17:23.396675 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:23.396500 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" podStartSLOduration=2.11524503 podStartE2EDuration="5.396480513s" totalImagesPullingTime="3.281235483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:23.395957759 +0000 UTC m=+666.208287936" watchObservedRunningTime="2026-11-02 22:17:23.396480513 +0000 UTC m=+666.208810693" Nov 02 22:17:24.296524 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:24.296473 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" event={"ID":"16b4519b-9df1-4c63-9f22-d46e804efef4","Type":"ContainerStarted","Data":"0bb1adef8fd6e98286a2f10cdfa65e1a5f9fcbbfed54021ae6c756987b8011a1"} Nov 02 22:17:24.297079 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:24.296730 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:24.439707 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:24.439601 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" podStartSLOduration=2.555662618 podStartE2EDuration="3.439583996s" totalImagesPullingTime="883.921378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:24.438039935 +0000 UTC m=+667.250370113" watchObservedRunningTime="2026-11-02 22:17:24.439583996 +0000 UTC m=+667.251914170" Nov 02 22:17:25.452334 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:25.452298 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:26.205013 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:26.204959 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:26.393563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:26.393520 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:27.609594 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:27.609539 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:28.105585 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:28.105538 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:28.286060 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:28.285970 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:29.826861 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:29.826824 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:30.309822 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:30.309793 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:17:32.203854 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.203817 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2"] Nov 02 22:17:32.206907 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.206863 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:32.257124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.257090 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-ca85db-serving-cert\"" Nov 02 22:17:32.287325 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.287283 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-metrics-certs\") pod \"maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:32.287563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.287388 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlm6l\" (UniqueName: \"kubernetes.io/projected/fd88af66-9693-4cd1-abd5-d2af89caa020-kube-api-access-wlm6l\") pod \"maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:32.287563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.287501 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-maas-api-tls\") pod \"maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:32.287651 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.287624 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd88af66-9693-4cd1-abd5-d2af89caa020-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:32.305732 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.305694 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2"] Nov 02 22:17:32.353742 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.353698 5871 generic.go:356] "Generic (PLEG): container finished" podID="fae99bb6-0aee-4ef6-a938-f189add936ed" containerID="aa30db04432008adadfc0c297ab3f6fce700209c999d2fec1716a46364a4e4b9" exitCode=0 Nov 02 22:17:32.353936 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.353746 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" event={"ID":"fae99bb6-0aee-4ef6-a938-f189add936ed","Type":"ContainerDied","Data":"aa30db04432008adadfc0c297ab3f6fce700209c999d2fec1716a46364a4e4b9"} Nov 02 22:17:32.354294 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.354269 5871 scope.go:122] "RemoveContainer" containerID="aa30db04432008adadfc0c297ab3f6fce700209c999d2fec1716a46364a4e4b9" Nov 02 22:17:32.355824 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.355792 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:32.355982 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.355895 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:32.355982 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.355948 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:17:32.375604 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.375573 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:17:32.388815 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.388777 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-maas-api-tls\") pod \"maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:32.389009 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.388851 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd88af66-9693-4cd1-abd5-d2af89caa020-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:32.389009 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:17:32.388939 5871 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-serving-cert: secret "maas-api-e2e-worker-w1-ca85db-serving-cert" not found Nov 02 22:17:32.389009 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.388995 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-metrics-certs\") pod \"maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:32.389192 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:17:32.389015 5871 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-maas-api-tls podName:fd88af66-9693-4cd1-abd5-d2af89caa020 nodeName:}" failed. No retries permitted until 2026-11-02 22:17:32.88899843 +0000 UTC m=+675.701328585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-maas-api-tls") pod "maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" (UID: "fd88af66-9693-4cd1-abd5-d2af89caa020") : secret "maas-api-e2e-worker-w1-ca85db-serving-cert" not found Nov 02 22:17:32.389192 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.389044 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlm6l\" (UniqueName: \"kubernetes.io/projected/fd88af66-9693-4cd1-abd5-d2af89caa020-kube-api-access-wlm6l\") pod \"maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:32.389488 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.389468 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd88af66-9693-4cd1-abd5-d2af89caa020-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:32.391409 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.391390 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-metrics-certs\") pod \"maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:32.440429 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.440403 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:32.446930 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.446900 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlm6l\" (UniqueName: \"kubernetes.io/projected/fd88af66-9693-4cd1-abd5-d2af89caa020-kube-api-access-wlm6l\") pod \"maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:32.616947 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.616868 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:32.804239 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.804201 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:32.899754 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.899643 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-maas-api-tls\") pod \"maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:32.902357 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:32.902326 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-maas-api-tls\") pod \"maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:33.119010 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:33.118964 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:33.386179 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:33.386145 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" event={"ID":"fae99bb6-0aee-4ef6-a938-f189add936ed","Type":"ContainerStarted","Data":"c2d217e5f1452decf2ef4f471490975ed9b146cb9873a2dc2e96be588acb917f"} Nov 02 22:17:33.528231 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:17:33.528193 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd88af66_9693_4cd1_abd5_d2af89caa020.slice/crio-34e3df37edb224135b28feec8ea8ed718923aa8fb6e54478a1d49ff570b8c25b WatchSource:0}: Error finding container 34e3df37edb224135b28feec8ea8ed718923aa8fb6e54478a1d49ff570b8c25b: Status 404 returned error can't find the container with id 34e3df37edb224135b28feec8ea8ed718923aa8fb6e54478a1d49ff570b8c25b Nov 02 22:17:33.547949 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:33.547908 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2"] Nov 02 22:17:33.725701 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:33.725667 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:17:33.726038 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:33.726014 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 22:17:33.726103 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:33.726064 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" podUID="fae99bb6-0aee-4ef6-a938-f189add936ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 22:17:34.299567 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:34.299534 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" Nov 02 22:17:34.392819 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:34.392782 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" event={"ID":"fd88af66-9693-4cd1-abd5-d2af89caa020","Type":"ContainerStarted","Data":"34e3df37edb224135b28feec8ea8ed718923aa8fb6e54478a1d49ff570b8c25b"} Nov 02 22:17:34.725492 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:34.725455 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 22:17:34.725683 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:34.725512 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" podUID="fae99bb6-0aee-4ef6-a938-f189add936ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 22:17:35.405155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:35.402923 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" event={"ID":"fd88af66-9693-4cd1-abd5-d2af89caa020","Type":"ContainerStarted","Data":"c0bf40f63c8a02aced0aa4292f4d2d059e671c5df724c928cef3a0eea59d964c"} Nov 02 22:17:35.405155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:35.403051 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:35.515120 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:35.515061 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" podStartSLOduration=2.544900845 podStartE2EDuration="3.515041858s" totalImagesPullingTime="970.141013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:35.512747086 +0000 UTC m=+678.325077282" watchObservedRunningTime="2026-11-02 22:17:35.515041858 +0000 UTC m=+678.327372118" Nov 02 22:17:35.725425 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:35.725332 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 22:17:35.725425 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:35.725388 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" podUID="fae99bb6-0aee-4ef6-a938-f189add936ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 22:17:36.250024 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:36.249971 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:36.725771 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:36.725720 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 22:17:36.726197 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:36.725785 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" podUID="fae99bb6-0aee-4ef6-a938-f189add936ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 22:17:36.858900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:36.853166 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:37.034667 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:37.034534 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:37.725782 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:37.725739 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 02 22:17:37.726194 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:37.725802 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" podUID="fae99bb6-0aee-4ef6-a938-f189add936ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 02 22:17:37.817088 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:37.816988 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:38.306110 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:38.306076 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:38.488396 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:38.488348 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:38.730955 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:38.730926 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:17:38.731405 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:38.731233 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:17:38.731944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:38.731926 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:17:40.234534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:40.234483 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:41.414052 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:41.414020 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:17:41.943389 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:41.943350 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:17:42.815366 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:42.815332 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:43.771106 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:17:43.771068 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:17:46.968238 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:46.968205 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:47.562920 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:47.562886 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:47.746919 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:47.746862 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:48.060467 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:48.060433 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:53.326926 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:53.326887 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:56.204510 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:56.204477 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:56.451470 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:56.451432 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:56.825303 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:56.825264 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:57.039193 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:57.039162 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:17:58.687248 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:17:58.687214 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:01.000046 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:01.000010 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:01.289256 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:01.289176 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:01.781859 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:01.781822 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:18:02.724574 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:02.724538 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:03.342001 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:03.341963 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:03.519335 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:03.519300 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:03.872428 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:03.872400 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:06.484988 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:06.484953 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:06.656613 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:06.656581 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:08.997226 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:08.997182 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:13.349603 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:13.349566 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:13.770061 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:18:13.770023 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:18:13.951069 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:13.951034 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:14.257312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:14.257256 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:16.179956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:16.179924 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:18:18.981462 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:18.981422 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:19.324304 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:19.324220 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:21.959682 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:21.959640 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:23.652936 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:23.652899 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:24.132520 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.132482 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:24.418483 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.418375 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9"] Nov 02 22:18:24.423392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.423371 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.474292 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.474259 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-49a557\"/\"premium-w6-49a557-kserve-self-signed-certs\"" Nov 02 22:18:24.498044 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.498004 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f259989-289c-4b0d-a158-9755a2ecb9fe-tls-certs\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.498207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.498117 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-tmp-dir\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.498207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.498146 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-dshm\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.498280 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.498206 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c95wk\" (UniqueName: \"kubernetes.io/projected/8f259989-289c-4b0d-a158-9755a2ecb9fe-kube-api-access-c95wk\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.498368 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.498349 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-home\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.498413 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.498400 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-model-cache\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.520494 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.520454 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-49a557\"/\"openshift-service-ca.crt\"" Nov 02 22:18:24.520494 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.520455 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-49a557\"/\"kube-root-ca.crt\"" Nov 02 22:18:24.522943 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.522923 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9"] Nov 02 22:18:24.538479 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.538423 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:24.564276 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.564240 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-49a557\"/\"default-dockercfg-bzn49\"" Nov 02 22:18:24.599617 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.599575 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f259989-289c-4b0d-a158-9755a2ecb9fe-tls-certs\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.599827 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.599653 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-tmp-dir\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.599827 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.599689 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-dshm\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.599827 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.599711 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c95wk\" (UniqueName: \"kubernetes.io/projected/8f259989-289c-4b0d-a158-9755a2ecb9fe-kube-api-access-c95wk\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.599827 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.599777 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-home\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.600086 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.599828 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-model-cache\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.600207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.600183 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-tmp-dir\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.600318 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.600300 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-model-cache\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.600377 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.600331 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-home\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.602129 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.602104 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-dshm\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.602367 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.602349 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f259989-289c-4b0d-a158-9755a2ecb9fe-tls-certs\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.657738 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.657705 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c95wk\" (UniqueName: \"kubernetes.io/projected/8f259989-289c-4b0d-a158-9755a2ecb9fe-kube-api-access-c95wk\") pod \"premium-w6-49a557-kserve-8f69bfbcf-644d9\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:24.735245 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:24.735148 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:25.125624 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:25.125582 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9"] Nov 02 22:18:25.131798 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:18:25.131771 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f259989_289c_4b0d_a158_9755a2ecb9fe.slice/crio-431a648041e122bbcccf12b4be073a3ab120d30c9278a7f89ef3b95149b37187 WatchSource:0}: Error finding container 431a648041e122bbcccf12b4be073a3ab120d30c9278a7f89ef3b95149b37187: Status 404 returned error can't find the container with id 431a648041e122bbcccf12b4be073a3ab120d30c9278a7f89ef3b95149b37187 Nov 02 22:18:25.711980 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:25.711938 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" event={"ID":"8f259989-289c-4b0d-a158-9755a2ecb9fe","Type":"ContainerStarted","Data":"483e2bd3c86e6a5cea18df1bef6197d4cb5da501332dac72c64310059c0a7c07"} Nov 02 22:18:25.712381 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:25.711992 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" event={"ID":"8f259989-289c-4b0d-a158-9755a2ecb9fe","Type":"ContainerStarted","Data":"431a648041e122bbcccf12b4be073a3ab120d30c9278a7f89ef3b95149b37187"} Nov 02 22:18:25.735654 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:25.735617 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:25.822888 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:25.822819 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podStartSLOduration=1.8228025140000002 podStartE2EDuration="1.822802514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:25.82062354 +0000 UTC m=+728.632953717" watchObservedRunningTime="2026-11-02 22:18:25.822802514 +0000 UTC m=+728.635132691" Nov 02 22:18:28.982677 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:28.982636 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:32.485154 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:32.485116 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:33.776636 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:33.776598 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:34.229611 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:34.229580 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:34.358589 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:34.358546 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:18:34.407038 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:34.407001 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:34.735887 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:34.735840 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:34.748679 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:34.748647 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:34.817574 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:34.817541 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:18:38.627826 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:38.627783 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:40.573548 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:40.573509 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:41.332252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.332221 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5"] Nov 02 22:18:41.336445 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.336421 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.433754 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.433721 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-98a222\"/\"kube-root-ca.crt\"" Nov 02 22:18:41.433754 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.433726 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-98a222\"/\"simulator-w3-98a222-kserve-self-signed-certs\"" Nov 02 22:18:41.433993 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.433780 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-98a222\"/\"openshift-service-ca.crt\"" Nov 02 22:18:41.433993 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.433982 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-98a222\"/\"default-dockercfg-9cwrj\"" Nov 02 22:18:41.436208 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.436190 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5"] Nov 02 22:18:41.450780 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.450748 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:41.479626 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.479591 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-model-cache\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.480776 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.479641 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-home\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.480776 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.479793 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c28tf\" (UniqueName: \"kubernetes.io/projected/d705b15d-5840-41da-b5d8-aad314aeb33b-kube-api-access-c28tf\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.480776 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.479826 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-dshm\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.480776 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.479996 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-tmp-dir\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.480776 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.480036 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d705b15d-5840-41da-b5d8-aad314aeb33b-tls-certs\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.581688 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.581646 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c28tf\" (UniqueName: \"kubernetes.io/projected/d705b15d-5840-41da-b5d8-aad314aeb33b-kube-api-access-c28tf\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.582223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.581694 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-dshm\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.582223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.581768 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-tmp-dir\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.582223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.581806 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d705b15d-5840-41da-b5d8-aad314aeb33b-tls-certs\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.582223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.581860 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-model-cache\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.582223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.581905 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-home\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.582587 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.582270 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-tmp-dir\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.582587 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.582312 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-model-cache\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.582587 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.582434 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-home\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.584589 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.584566 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-dshm\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.584964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.584945 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d705b15d-5840-41da-b5d8-aad314aeb33b-tls-certs\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.638868 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.638827 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c28tf\" (UniqueName: \"kubernetes.io/projected/d705b15d-5840-41da-b5d8-aad314aeb33b-kube-api-access-c28tf\") pod \"simulator-w3-98a222-kserve-749cbbb68-zpfs5\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:41.649704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:41.649667 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:42.029193 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:42.029151 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5"] Nov 02 22:18:42.035697 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:18:42.035668 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd705b15d_5840_41da_b5d8_aad314aeb33b.slice/crio-199e6a737a31ab3f598396bbed39e5f6d035378cf86406c4236e122271f8670a WatchSource:0}: Error finding container 199e6a737a31ab3f598396bbed39e5f6d035378cf86406c4236e122271f8670a: Status 404 returned error can't find the container with id 199e6a737a31ab3f598396bbed39e5f6d035378cf86406c4236e122271f8670a Nov 02 22:18:42.852680 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:42.852637 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" event={"ID":"d705b15d-5840-41da-b5d8-aad314aeb33b","Type":"ContainerStarted","Data":"795daadd62dd8ecc61b165629aa0c8d2ece2503d52dce29b6605f1ecb5c8510e"} Nov 02 22:18:42.853263 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:42.852694 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" event={"ID":"d705b15d-5840-41da-b5d8-aad314aeb33b","Type":"ContainerStarted","Data":"199e6a737a31ab3f598396bbed39e5f6d035378cf86406c4236e122271f8670a"} Nov 02 22:18:42.964888 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:42.964809 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podStartSLOduration=1.964794097 podStartE2EDuration="1.964794097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:42.96238083 +0000 UTC m=+745.774711009" watchObservedRunningTime="2026-11-02 22:18:42.964794097 +0000 UTC m=+745.777124275" Nov 02 22:18:43.499599 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:43.499558 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:43.650036 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:43.650005 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:43.771164 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:18:43.771074 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:18:48.230227 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:48.230168 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:50.589145 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:50.589103 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:18:51.650486 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:51.650448 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:51.663237 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:51.663203 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:51.917867 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:51.917777 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:18:53.071228 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:53.071164 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:57.823865 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:57.823823 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:18:58.695694 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:18:58.695650 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:01.356621 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:01.356587 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:02.669296 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:02.669237 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:04.870379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:04.870340 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:19:07.579163 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:07.579103 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:09.516501 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:09.516442 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:12.394740 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:12.394702 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:13.771489 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:19:13.771453 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:19:16.011399 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.011355 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv"] Nov 02 22:19:16.015768 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.015741 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.067534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.067501 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ca85db\"/\"openshift-service-ca.crt\"" Nov 02 22:19:16.113904 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.113860 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ca85db\"/\"default-dockercfg-5tbpv\"" Nov 02 22:19:16.114088 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.113988 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ca85db\"/\"kube-root-ca.crt\"" Nov 02 22:19:16.114088 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.114063 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ca85db\"/\"simulator-w1-ca85db-kserve-self-signed-certs\"" Nov 02 22:19:16.118691 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.118658 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv"] Nov 02 22:19:16.131835 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.131799 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:16.182393 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.182363 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-model-cache\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.182564 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.182401 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/180c8830-c2d7-42b4-9a5e-41f15996a8d6-tls-certs\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.182564 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.182520 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-home\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.182641 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.182613 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-dshm\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.182692 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.182676 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-tmp-dir\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.182732 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.182710 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mszns\" (UniqueName: \"kubernetes.io/projected/180c8830-c2d7-42b4-9a5e-41f15996a8d6-kube-api-access-mszns\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.284337 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.284243 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-model-cache\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.284337 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.284292 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/180c8830-c2d7-42b4-9a5e-41f15996a8d6-tls-certs\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.284337 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.284323 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-home\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.284624 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.284360 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-dshm\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.284624 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.284414 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-tmp-dir\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.284624 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.284437 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mszns\" (UniqueName: \"kubernetes.io/projected/180c8830-c2d7-42b4-9a5e-41f15996a8d6-kube-api-access-mszns\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.284793 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.284716 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-model-cache\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.284793 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.284764 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-home\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.284935 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.284831 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-tmp-dir\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.286728 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.286696 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-dshm\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.286915 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.286897 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/180c8830-c2d7-42b4-9a5e-41f15996a8d6-tls-certs\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.342557 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.342515 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mszns\" (UniqueName: \"kubernetes.io/projected/180c8830-c2d7-42b4-9a5e-41f15996a8d6-kube-api-access-mszns\") pod \"simulator-w1-ca85db-kserve-77795c999-mnstv\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:16.628084 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:16.628039 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:17.018052 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:17.017946 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv"] Nov 02 22:19:17.023649 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:19:17.023619 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod180c8830_c2d7_42b4_9a5e_41f15996a8d6.slice/crio-4c2f3d3c77db39d9dd754e3adfbb65248e9ec83e3cb1c8c957be5beadf8e7f13 WatchSource:0}: Error finding container 4c2f3d3c77db39d9dd754e3adfbb65248e9ec83e3cb1c8c957be5beadf8e7f13: Status 404 returned error can't find the container with id 4c2f3d3c77db39d9dd754e3adfbb65248e9ec83e3cb1c8c957be5beadf8e7f13 Nov 02 22:19:17.065327 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:17.065282 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" event={"ID":"180c8830-c2d7-42b4-9a5e-41f15996a8d6","Type":"ContainerStarted","Data":"4c2f3d3c77db39d9dd754e3adfbb65248e9ec83e3cb1c8c957be5beadf8e7f13"} Nov 02 22:19:17.210555 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:17.210495 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:18.073166 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:18.073119 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" event={"ID":"180c8830-c2d7-42b4-9a5e-41f15996a8d6","Type":"ContainerStarted","Data":"c54b7568f10299d80d9027850cf09d654ae157a0af94ee39c434140c53ca32a8"} Nov 02 22:19:18.185334 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:18.185276 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podStartSLOduration=3.185262339 podStartE2EDuration="3.185262339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:18.18320553 +0000 UTC m=+780.995535707" watchObservedRunningTime="2026-11-02 22:19:18.185262339 +0000 UTC m=+780.997592621" Nov 02 22:19:18.628177 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:18.628137 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:22.087458 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:22.087381 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:22.310899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:22.310805 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:26.629070 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:26.629038 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:26.641579 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:26.641552 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:26.886623 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:26.886525 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:27.139455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:27.139368 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:19:31.191343 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:31.191287 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:31.653727 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:31.653674 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:35.506918 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:35.506863 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:19:36.576271 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:36.576236 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:40.928905 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:40.928833 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:41.516821 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:41.516782 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:43.771219 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:19:43.771180 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:19:46.479613 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:46.479577 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:51.281122 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:51.281089 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:54.650516 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.650424 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p"] Nov 02 22:19:54.655097 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.655074 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.704328 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.704289 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-tmp-dir\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.704488 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.704339 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-dshm\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.704488 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.704458 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-tls-certs\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.704659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.704642 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr2v4\" (UniqueName: \"kubernetes.io/projected/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-kube-api-access-kr2v4\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.704715 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.704699 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-model-cache\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.704754 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.704733 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-home\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.719639 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.719606 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p"] Nov 02 22:19:54.752545 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.752513 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8263a4\"/\"kube-root-ca.crt\"" Nov 02 22:19:54.752545 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.752538 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8263a4\"/\"premium-w2-8263a4-kserve-self-signed-certs\"" Nov 02 22:19:54.752545 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.752513 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8263a4\"/\"openshift-service-ca.crt\"" Nov 02 22:19:54.752774 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.752517 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8263a4\"/\"default-dockercfg-bj2sh\"" Nov 02 22:19:54.770989 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.770959 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:54.806433 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.806391 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2v4\" (UniqueName: \"kubernetes.io/projected/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-kube-api-access-kr2v4\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.806647 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.806476 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-model-cache\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.806647 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.806512 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-home\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.806647 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.806562 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-tmp-dir\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.806647 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.806601 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-dshm\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.806647 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.806641 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-tls-certs\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.807318 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.807269 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-model-cache\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.808011 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.807744 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-tmp-dir\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.808231 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.808094 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-home\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.811921 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.810800 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-dshm\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.812716 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.812696 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-tls-certs\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.864132 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.864095 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr2v4\" (UniqueName: \"kubernetes.io/projected/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-kube-api-access-kr2v4\") pod \"premium-w2-8263a4-kserve-7fd5b4d864-vsc2p\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:54.990944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:54.990812 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:19:55.380657 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:55.380620 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p"] Nov 02 22:19:55.386575 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:19:55.386547 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1d285f6_5118_41bc_aa3a_0c3348c8d1b6.slice/crio-50b1592c196264226e8f52c2b6ba261bafad275339c63948f0143d61fab57e26 WatchSource:0}: Error finding container 50b1592c196264226e8f52c2b6ba261bafad275339c63948f0143d61fab57e26: Status 404 returned error can't find the container with id 50b1592c196264226e8f52c2b6ba261bafad275339c63948f0143d61fab57e26 Nov 02 22:19:56.309088 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:56.309051 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" event={"ID":"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6","Type":"ContainerStarted","Data":"7a9de685be59f2fbcedf0a63f9228f69e803cafea059231566802828ebe114d6"} Nov 02 22:19:56.309088 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:56.309094 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" event={"ID":"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6","Type":"ContainerStarted","Data":"50b1592c196264226e8f52c2b6ba261bafad275339c63948f0143d61fab57e26"} Nov 02 22:19:56.423613 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:56.422715 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podStartSLOduration=2.422695804 podStartE2EDuration="2.422695804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:56.417966282 +0000 UTC m=+819.230296459" watchObservedRunningTime="2026-11-02 22:19:56.422695804 +0000 UTC m=+819.235025983" Nov 02 22:19:56.557421 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:56.557383 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:19:56.991677 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:19:56.991636 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:20:00.968304 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:00.968268 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:01.048596 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:01.048557 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:20:01.726065 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:01.726019 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:04.991503 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:04.991463 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:20:05.004449 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:05.004420 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:20:05.375799 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:05.375771 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:20:05.876987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:05.876950 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:08.080087 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:08.080050 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:09.634849 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:09.634811 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:10.784569 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:10.784532 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:13.770539 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:20:13.770495 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:20:15.584611 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:15.584574 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:20.484214 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:20.484167 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:22.641898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:22.641842 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:25.385614 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:25.385580 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:30.188253 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:30.188210 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:34.483329 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:34.483289 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:20:35.078563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:35.078524 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:35.274128 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:35.274091 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:38.964895 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:38.964841 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6"] Nov 02 22:20:38.969128 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:38.969102 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.018197 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.018164 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-98a222-openshift-default-dockercfg-hhv4j\"" Nov 02 22:20:39.068074 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.068040 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6"] Nov 02 22:20:39.104257 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.104221 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.104257 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.104264 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.104496 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.104343 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667069a7-93cf-42ef-92ec-e825e6f6fabd-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.104496 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.104388 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.104496 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.104427 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.104639 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.104498 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czxhw\" (UniqueName: \"kubernetes.io/projected/667069a7-93cf-42ef-92ec-e825e6f6fabd-kube-api-access-czxhw\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.104639 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.104624 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.104732 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.104719 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.104767 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.104742 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.205807 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.205770 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.205807 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.205817 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czxhw\" (UniqueName: \"kubernetes.io/projected/667069a7-93cf-42ef-92ec-e825e6f6fabd-kube-api-access-czxhw\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.206100 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.205856 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.206100 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.205979 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.206213 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.206109 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.206213 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.206185 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.206322 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.206221 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.206322 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.206267 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.206322 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.206287 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667069a7-93cf-42ef-92ec-e825e6f6fabd-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.206487 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.206364 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.206487 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.206405 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.206584 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.206502 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.206584 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.206532 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.207091 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.207067 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667069a7-93cf-42ef-92ec-e825e6f6fabd-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.208454 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.208433 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.208562 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.208478 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.264994 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.264912 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.264994 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.264981 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czxhw\" (UniqueName: \"kubernetes.io/projected/667069a7-93cf-42ef-92ec-e825e6f6fabd-kube-api-access-czxhw\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.282850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.282819 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:39.715852 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.715816 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6"] Nov 02 22:20:39.722406 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:20:39.722382 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667069a7_93cf_42ef_92ec_e825e6f6fabd.slice/crio-437386da1579491052b58192748c4be6d87b8d3544a5514005b2c74782a21234 WatchSource:0}: Error finding container 437386da1579491052b58192748c4be6d87b8d3544a5514005b2c74782a21234: Status 404 returned error can't find the container with id 437386da1579491052b58192748c4be6d87b8d3544a5514005b2c74782a21234 Nov 02 22:20:39.724827 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.724796 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:20:39.724941 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.724866 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:20:39.724941 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:39.724914 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:20:40.008609 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:40.008516 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:40.590399 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:40.590360 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" event={"ID":"667069a7-93cf-42ef-92ec-e825e6f6fabd","Type":"ContainerStarted","Data":"4f43ede9421c2070255dbf751f1e3b35ef9cdb63ec292b044d570891613817ad"} Nov 02 22:20:40.590621 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:40.590411 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" event={"ID":"667069a7-93cf-42ef-92ec-e825e6f6fabd","Type":"ContainerStarted","Data":"437386da1579491052b58192748c4be6d87b8d3544a5514005b2c74782a21234"} Nov 02 22:20:40.798638 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:40.798579 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" podStartSLOduration=2.798562206 podStartE2EDuration="2.798562206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:40.749817196 +0000 UTC m=+863.562147374" watchObservedRunningTime="2026-11-02 22:20:40.798562206 +0000 UTC m=+863.610892385" Nov 02 22:20:41.282476 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:41.282435 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:41.283297 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:41.283269 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:41.284778 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:41.284755 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 22:20:41.284886 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:41.284812 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" podUID="667069a7-93cf-42ef-92ec-e825e6f6fabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 22:20:42.283474 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:42.283436 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 22:20:42.283907 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:42.283529 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" podUID="667069a7-93cf-42ef-92ec-e825e6f6fabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 22:20:43.283886 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:43.283835 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 22:20:43.284290 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:43.283940 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" podUID="667069a7-93cf-42ef-92ec-e825e6f6fabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 22:20:43.790351 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:20:43.790317 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:20:44.283913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:44.283860 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 22:20:44.284312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:44.283943 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" podUID="667069a7-93cf-42ef-92ec-e825e6f6fabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 22:20:44.790350 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:44.790315 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:45.283301 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:45.283267 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 22:20:45.283502 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:45.283340 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" podUID="667069a7-93cf-42ef-92ec-e825e6f6fabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 22:20:46.454039 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:46.454002 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:46.454809 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:46.454780 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:46.455918 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:46.455863 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:20:49.589938 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:49.589901 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:52.749880 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.749830 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br"] Nov 02 22:20:52.753885 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.753854 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.802705 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.802672 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-711bd4-openshift-default-dockercfg-h26gm\"" Nov 02 22:20:52.851802 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.851765 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br"] Nov 02 22:20:52.884222 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.884173 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c910d204-821e-4a39-9f51-40668aaa2e7e-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.884429 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.884346 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.884429 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.884402 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.884515 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.884469 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.884515 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.884504 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.884594 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.884580 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.884638 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.884608 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.884697 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.884635 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5hrk\" (UniqueName: \"kubernetes.io/projected/c910d204-821e-4a39-9f51-40668aaa2e7e-kube-api-access-t5hrk\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.884751 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.884720 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.986049 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.986011 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.986049 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.986054 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.986300 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.986078 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5hrk\" (UniqueName: \"kubernetes.io/projected/c910d204-821e-4a39-9f51-40668aaa2e7e-kube-api-access-t5hrk\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.986300 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.986210 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.986300 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.986276 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c910d204-821e-4a39-9f51-40668aaa2e7e-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.986461 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.986428 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.986526 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.986492 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.986526 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.986498 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.986629 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.986529 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.986629 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.986577 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.986798 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.986776 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.986995 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.986960 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.987115 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.987033 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.987115 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.987062 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c910d204-821e-4a39-9f51-40668aaa2e7e-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.988527 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.988503 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:52.988753 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:52.988735 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:53.045831 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:53.045757 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:53.045990 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:53.045913 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5hrk\" (UniqueName: \"kubernetes.io/projected/c910d204-821e-4a39-9f51-40668aaa2e7e-kube-api-access-t5hrk\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:53.068789 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:53.068754 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:53.502616 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:53.502579 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br"] Nov 02 22:20:53.508648 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:20:53.508610 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc910d204_821e_4a39_9f51_40668aaa2e7e.slice/crio-c4f8dc3e359a9d1d9fb3ba089f9569bb6d1430595d94b1259a8ed0d3b0db0214 WatchSource:0}: Error finding container c4f8dc3e359a9d1d9fb3ba089f9569bb6d1430595d94b1259a8ed0d3b0db0214: Status 404 returned error can't find the container with id c4f8dc3e359a9d1d9fb3ba089f9569bb6d1430595d94b1259a8ed0d3b0db0214 Nov 02 22:20:53.511134 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:53.511103 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:20:53.511210 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:53.511170 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:20:53.511210 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:53.511204 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:20:53.684338 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:53.684298 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" event={"ID":"c910d204-821e-4a39-9f51-40668aaa2e7e","Type":"ContainerStarted","Data":"04685e926389071813a0c82cea20a192bd9671faa5e5fdd67091038fd6668b18"} Nov 02 22:20:53.684516 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:53.684362 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" event={"ID":"c910d204-821e-4a39-9f51-40668aaa2e7e","Type":"ContainerStarted","Data":"c4f8dc3e359a9d1d9fb3ba089f9569bb6d1430595d94b1259a8ed0d3b0db0214"} Nov 02 22:20:53.800297 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:53.800171 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" podStartSLOduration=1.800157205 podStartE2EDuration="1.800157205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:53.796950799 +0000 UTC m=+876.609280977" watchObservedRunningTime="2026-11-02 22:20:53.800157205 +0000 UTC m=+876.612487382" Nov 02 22:20:54.069309 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:54.069219 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:20:54.070570 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:54.070548 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 22:20:54.070629 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:54.070599 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" podUID="c910d204-821e-4a39-9f51-40668aaa2e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 22:20:54.375904 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:54.375844 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:20:55.073897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:55.073434 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 22:20:55.073897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:55.073527 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" podUID="c910d204-821e-4a39-9f51-40668aaa2e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 22:20:56.078444 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:56.078402 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 22:20:56.078923 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:56.078471 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" podUID="c910d204-821e-4a39-9f51-40668aaa2e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 22:20:57.071356 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:57.071312 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 22:20:57.072222 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:57.072188 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" podUID="c910d204-821e-4a39-9f51-40668aaa2e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 22:20:57.195452 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:57.195413 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv"] Nov 02 22:20:57.195921 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:57.195647 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" podUID="88a69dbf-afb3-4e8d-ac96-2207095ca1c4" containerName="istio-proxy" containerID="cri-o://82eeaf7a11fed848a1a90fe82a065af24fc28c312e194725423d26e5e7456793" gracePeriod=30 Nov 02 22:20:57.241556 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:57.241519 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x"] Nov 02 22:20:57.241741 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:57.241721 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" podUID="4e6f0006-1702-4d03-89e5-8ed60a8203ee" containerName="istio-proxy" containerID="cri-o://852d7f3a7754e9c35abbe6b4a8fa82817400b83e0fee63e1e0101a064ada5d01" gracePeriod=30 Nov 02 22:20:58.069357 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:58.069313 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 22:20:58.069676 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:58.069646 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" podUID="c910d204-821e-4a39-9f51-40668aaa2e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 22:20:59.073323 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:59.073285 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 22:20:59.073956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:59.073924 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" podUID="c910d204-821e-4a39-9f51-40668aaa2e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 22:20:59.421804 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:20:59.421767 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:00.070215 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:00.070176 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 22:21:00.070406 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:00.070238 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" podUID="c910d204-821e-4a39-9f51-40668aaa2e7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 22:21:01.074068 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:01.074037 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:21:01.074484 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:01.074463 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:21:01.075112 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:01.075091 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:21:02.093335 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.093299 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:02.508858 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.508830 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:21:02.531972 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.531946 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:21:02.632794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.632763 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-kube-api-access-ltd9w\" (UniqueName: \"kubernetes.io/projected/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-kube-api-access-ltd9w\") pod \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " Nov 02 22:21:02.632973 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.632808 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istiod-ca-cert\") pod \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " Nov 02 22:21:02.632973 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.632846 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-credential-socket\") pod \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " Nov 02 22:21:02.632973 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.632951 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-credential-socket\") pod \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " Nov 02 22:21:02.633198 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.632985 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-workload-socket\") pod \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " Nov 02 22:21:02.633198 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633037 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-podinfo\") pod \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " Nov 02 22:21:02.633198 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633077 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-data\") pod \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " Nov 02 22:21:02.633198 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633127 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-token\" (UniqueName: \"kubernetes.io/projected/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-token\") pod \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " Nov 02 22:21:02.633198 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633167 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-envoy\") pod \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " Nov 02 22:21:02.633198 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633192 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-podinfo\") pod \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " Nov 02 22:21:02.633515 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633244 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-workload-certs\") pod \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " Nov 02 22:21:02.633515 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633275 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istiod-ca-cert\") pod \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " Nov 02 22:21:02.633515 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633305 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-data\") pod \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " Nov 02 22:21:02.633515 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633311 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istiod-ca-cert" pod "88a69dbf-afb3-4e8d-ac96-2207095ca1c4" (UID: "88a69dbf-afb3-4e8d-ac96-2207095ca1c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:02.633515 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633350 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-envoy\") pod \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " Nov 02 22:21:02.633515 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633387 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-workload-certs\") pod \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " Nov 02 22:21:02.633515 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633433 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-token\" (UniqueName: \"kubernetes.io/projected/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-token\") pod \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " Nov 02 22:21:02.633515 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633468 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6f0006-1702-4d03-89e5-8ed60a8203ee-kube-api-access-h59fv\" (UniqueName: \"kubernetes.io/projected/4e6f0006-1702-4d03-89e5-8ed60a8203ee-kube-api-access-h59fv\") pod \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\" (UID: \"4e6f0006-1702-4d03-89e5-8ed60a8203ee\") " Nov 02 22:21:02.633515 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633469 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-credential-socket" pod "88a69dbf-afb3-4e8d-ac96-2207095ca1c4" (UID: "88a69dbf-afb3-4e8d-ac96-2207095ca1c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:02.633515 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633478 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-workload-socket" pod "4e6f0006-1702-4d03-89e5-8ed60a8203ee" (UID: "4e6f0006-1702-4d03-89e5-8ed60a8203ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:02.634154 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633583 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-workload-socket\") pod \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\" (UID: \"88a69dbf-afb3-4e8d-ac96-2207095ca1c4\") " Nov 02 22:21:02.634154 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633697 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-workload-certs" pod "88a69dbf-afb3-4e8d-ac96-2207095ca1c4" (UID: "88a69dbf-afb3-4e8d-ac96-2207095ca1c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:02.634154 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633767 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-data" pod "4e6f0006-1702-4d03-89e5-8ed60a8203ee" (UID: "4e6f0006-1702-4d03-89e5-8ed60a8203ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:02.634154 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633802 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-credential-socket" pod "4e6f0006-1702-4d03-89e5-8ed60a8203ee" (UID: "4e6f0006-1702-4d03-89e5-8ed60a8203ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:02.634154 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.633915 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-data" pod "88a69dbf-afb3-4e8d-ac96-2207095ca1c4" (UID: "88a69dbf-afb3-4e8d-ac96-2207095ca1c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:02.634154 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.634122 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istiod-ca-cert" pod "4e6f0006-1702-4d03-89e5-8ed60a8203ee" (UID: "4e6f0006-1702-4d03-89e5-8ed60a8203ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:02.634154 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.634148 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.634543 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.634172 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.634543 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.634187 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.634543 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.634202 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.634543 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.634216 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.634543 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.634228 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.634543 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.634241 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.634543 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.634478 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-workload-certs" pod "4e6f0006-1702-4d03-89e5-8ed60a8203ee" (UID: "4e6f0006-1702-4d03-89e5-8ed60a8203ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:02.634906 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.634770 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-workload-socket" pod "88a69dbf-afb3-4e8d-ac96-2207095ca1c4" (UID: "88a69dbf-afb3-4e8d-ac96-2207095ca1c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:02.636859 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.636830 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-envoy" pod "4e6f0006-1702-4d03-89e5-8ed60a8203ee" (UID: "4e6f0006-1702-4d03-89e5-8ed60a8203ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:02.637102 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.637076 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-envoy" pod "88a69dbf-afb3-4e8d-ac96-2207095ca1c4" (UID: "88a69dbf-afb3-4e8d-ac96-2207095ca1c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:21:02.637238 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.637202 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-podinfo" pod "4e6f0006-1702-4d03-89e5-8ed60a8203ee" (UID: "4e6f0006-1702-4d03-89e5-8ed60a8203ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:21:02.637238 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.637230 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-kube-api-access-ltd9w" pod "88a69dbf-afb3-4e8d-ac96-2207095ca1c4" (UID: "88a69dbf-afb3-4e8d-ac96-2207095ca1c4"). InnerVolumeSpecName "kube-api-access-ltd9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:02.637362 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.637303 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-token" pod "88a69dbf-afb3-4e8d-ac96-2207095ca1c4" (UID: "88a69dbf-afb3-4e8d-ac96-2207095ca1c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:02.638062 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.638036 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-podinfo" pod "88a69dbf-afb3-4e8d-ac96-2207095ca1c4" (UID: "88a69dbf-afb3-4e8d-ac96-2207095ca1c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:21:02.638166 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.638094 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6f0006-1702-4d03-89e5-8ed60a8203ee-kube-api-access-h59fv" pod "4e6f0006-1702-4d03-89e5-8ed60a8203ee" (UID: "4e6f0006-1702-4d03-89e5-8ed60a8203ee"). InnerVolumeSpecName "kube-api-access-h59fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:02.638442 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.638416 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-token" pod "4e6f0006-1702-4d03-89e5-8ed60a8203ee" (UID: "4e6f0006-1702-4d03-89e5-8ed60a8203ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:02.734915 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.734853 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.734915 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.734916 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltd9w\" (UniqueName: \"kubernetes.io/projected/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-kube-api-access-ltd9w\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.735144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.734934 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.735144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.734948 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.735144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.734960 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.735144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.734972 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.735144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.734983 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.735144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.734997 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.735144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.735009 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e6f0006-1702-4d03-89e5-8ed60a8203ee-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.735144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.735020 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a69dbf-afb3-4e8d-ac96-2207095ca1c4-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.735144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.735033 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h59fv\" (UniqueName: \"kubernetes.io/projected/4e6f0006-1702-4d03-89e5-8ed60a8203ee-kube-api-access-h59fv\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:02.744056 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.744024 5871 generic.go:356] "Generic (PLEG): container finished" podID="88a69dbf-afb3-4e8d-ac96-2207095ca1c4" containerID="82eeaf7a11fed848a1a90fe82a065af24fc28c312e194725423d26e5e7456793" exitCode=0 Nov 02 22:21:02.744207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.744073 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" Nov 02 22:21:02.744207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.744106 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" event={"ID":"88a69dbf-afb3-4e8d-ac96-2207095ca1c4","Type":"ContainerDied","Data":"82eeaf7a11fed848a1a90fe82a065af24fc28c312e194725423d26e5e7456793"} Nov 02 22:21:02.744207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.744164 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv" event={"ID":"88a69dbf-afb3-4e8d-ac96-2207095ca1c4","Type":"ContainerDied","Data":"84c2e8b0217975fb4d3cb2ced9bfb04a1125b0b92976660ab06a45acecb34291"} Nov 02 22:21:02.744207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.744187 5871 scope.go:122] "RemoveContainer" containerID="82eeaf7a11fed848a1a90fe82a065af24fc28c312e194725423d26e5e7456793" Nov 02 22:21:02.745851 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.745825 5871 generic.go:356] "Generic (PLEG): container finished" podID="4e6f0006-1702-4d03-89e5-8ed60a8203ee" containerID="852d7f3a7754e9c35abbe6b4a8fa82817400b83e0fee63e1e0101a064ada5d01" exitCode=0 Nov 02 22:21:02.746602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.746585 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" Nov 02 22:21:02.750964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.750933 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" event={"ID":"4e6f0006-1702-4d03-89e5-8ed60a8203ee","Type":"ContainerDied","Data":"852d7f3a7754e9c35abbe6b4a8fa82817400b83e0fee63e1e0101a064ada5d01"} Nov 02 22:21:02.751052 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.750991 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x" event={"ID":"4e6f0006-1702-4d03-89e5-8ed60a8203ee","Type":"ContainerDied","Data":"6def666cd381cff55fe911cb8e0b3159bf27ec0e82b08cf310dba5a976a8a68a"} Nov 02 22:21:02.754345 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.754317 5871 scope.go:122] "RemoveContainer" containerID="82eeaf7a11fed848a1a90fe82a065af24fc28c312e194725423d26e5e7456793" Nov 02 22:21:02.754763 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:21:02.754624 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82eeaf7a11fed848a1a90fe82a065af24fc28c312e194725423d26e5e7456793\": container with ID starting with 82eeaf7a11fed848a1a90fe82a065af24fc28c312e194725423d26e5e7456793 not found: ID does not exist" containerID="82eeaf7a11fed848a1a90fe82a065af24fc28c312e194725423d26e5e7456793" Nov 02 22:21:02.754763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.754661 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82eeaf7a11fed848a1a90fe82a065af24fc28c312e194725423d26e5e7456793"} err="failed to get container status \"82eeaf7a11fed848a1a90fe82a065af24fc28c312e194725423d26e5e7456793\": rpc error: code = NotFound desc = could not find container \"82eeaf7a11fed848a1a90fe82a065af24fc28c312e194725423d26e5e7456793\": container with ID starting with 82eeaf7a11fed848a1a90fe82a065af24fc28c312e194725423d26e5e7456793 not found: ID does not exist" Nov 02 22:21:02.754763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.754689 5871 scope.go:122] "RemoveContainer" containerID="852d7f3a7754e9c35abbe6b4a8fa82817400b83e0fee63e1e0101a064ada5d01" Nov 02 22:21:02.764367 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.764347 5871 scope.go:122] "RemoveContainer" containerID="852d7f3a7754e9c35abbe6b4a8fa82817400b83e0fee63e1e0101a064ada5d01" Nov 02 22:21:02.764659 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:21:02.764641 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"852d7f3a7754e9c35abbe6b4a8fa82817400b83e0fee63e1e0101a064ada5d01\": container with ID starting with 852d7f3a7754e9c35abbe6b4a8fa82817400b83e0fee63e1e0101a064ada5d01 not found: ID does not exist" containerID="852d7f3a7754e9c35abbe6b4a8fa82817400b83e0fee63e1e0101a064ada5d01" Nov 02 22:21:02.764701 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.764670 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"852d7f3a7754e9c35abbe6b4a8fa82817400b83e0fee63e1e0101a064ada5d01"} err="failed to get container status \"852d7f3a7754e9c35abbe6b4a8fa82817400b83e0fee63e1e0101a064ada5d01\": rpc error: code = NotFound desc = could not find container \"852d7f3a7754e9c35abbe6b4a8fa82817400b83e0fee63e1e0101a064ada5d01\": container with ID starting with 852d7f3a7754e9c35abbe6b4a8fa82817400b83e0fee63e1e0101a064ada5d01 not found: ID does not exist" Nov 02 22:21:02.886116 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.886033 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x"] Nov 02 22:21:02.892441 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.892413 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-6vm9x"] Nov 02 22:21:02.995766 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:02.995729 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv"] Nov 02 22:21:03.001548 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:03.001521 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-9jnjv"] Nov 02 22:21:03.313373 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:03.313343 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e6f0006-1702-4d03-89e5-8ed60a8203ee" path="/var/lib/kubelet/pods/4e6f0006-1702-4d03-89e5-8ed60a8203ee/volumes" Nov 02 22:21:03.313809 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:03.313794 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a69dbf-afb3-4e8d-ac96-2207095ca1c4" path="/var/lib/kubelet/pods/88a69dbf-afb3-4e8d-ac96-2207095ca1c4/volumes" Nov 02 22:21:04.281989 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:04.281918 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:06.333961 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:06.333913 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:06.511190 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:06.511138 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:09.103005 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:09.102946 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:13.770740 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:21:13.770705 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:21:14.020863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:14.020763 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:18.883314 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:18.883275 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:18.971563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:18.971520 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:21:20.208590 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:20.208503 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:20.385808 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:20.385771 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:23.057055 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:23.057013 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:23.894463 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:23.894425 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:28.760598 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:28.760495 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:33.655259 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:33.655224 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:35.994240 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:35.994205 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb"] Nov 02 22:21:35.994706 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:35.994683 5871 cpu_manager.go:414] "containerMap: removing container" podUID="4e6f0006-1702-4d03-89e5-8ed60a8203ee" containerName="istio-proxy" Nov 02 22:21:35.994706 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:35.994699 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e6f0006-1702-4d03-89e5-8ed60a8203ee" containerName="istio-proxy" Nov 02 22:21:35.994884 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:35.994724 5871 cpu_manager.go:414] "containerMap: removing container" podUID="88a69dbf-afb3-4e8d-ac96-2207095ca1c4" containerName="istio-proxy" Nov 02 22:21:35.994884 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:35.994730 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a69dbf-afb3-4e8d-ac96-2207095ca1c4" containerName="istio-proxy" Nov 02 22:21:35.994884 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:35.994810 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e6f0006-1702-4d03-89e5-8ed60a8203ee" containerName="istio-proxy" Nov 02 22:21:35.994884 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:35.994821 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a69dbf-afb3-4e8d-ac96-2207095ca1c4" containerName="istio-proxy" Nov 02 22:21:35.998210 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:35.998188 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.064363 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.064327 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb"] Nov 02 22:21:36.095625 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.095591 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-98a222\"/\"distinct-w3-98a222-kserve-self-signed-certs\"" Nov 02 22:21:36.111233 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.111198 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:36.114471 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.114442 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbpkc\" (UniqueName: \"kubernetes.io/projected/36cf5632-ebba-49b5-ad2c-40f5004570bd-kube-api-access-mbpkc\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.114621 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.114497 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-dshm\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.114621 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.114611 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-tmp-dir\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.114731 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.114665 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-home\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.114774 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.114756 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36cf5632-ebba-49b5-ad2c-40f5004570bd-tls-certs\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.114840 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.114826 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-model-cache\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.216139 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.216092 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbpkc\" (UniqueName: \"kubernetes.io/projected/36cf5632-ebba-49b5-ad2c-40f5004570bd-kube-api-access-mbpkc\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.216331 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.216153 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-dshm\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.216331 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.216183 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-tmp-dir\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.216331 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.216214 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-home\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.216492 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.216339 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36cf5632-ebba-49b5-ad2c-40f5004570bd-tls-certs\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.216492 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.216394 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-model-cache\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.216659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.216640 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-home\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.216746 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.216723 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-tmp-dir\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.216793 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.216769 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-model-cache\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.218649 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.218627 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-dshm\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.219062 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.219040 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36cf5632-ebba-49b5-ad2c-40f5004570bd-tls-certs\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.309906 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.309783 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbpkc\" (UniqueName: \"kubernetes.io/projected/36cf5632-ebba-49b5-ad2c-40f5004570bd-kube-api-access-mbpkc\") pod \"distinct-w3-98a222-kserve-65475c68cb-fsltb\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.310545 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.310518 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:36.702851 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.702815 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb"] Nov 02 22:21:36.709279 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:21:36.709240 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36cf5632_ebba_49b5_ad2c_40f5004570bd.slice/crio-29be60302f21def038fedf9c1347ce7f7fad555191f12f1883f4ac734a6df0d2 WatchSource:0}: Error finding container 29be60302f21def038fedf9c1347ce7f7fad555191f12f1883f4ac734a6df0d2: Status 404 returned error can't find the container with id 29be60302f21def038fedf9c1347ce7f7fad555191f12f1883f4ac734a6df0d2 Nov 02 22:21:36.970109 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.970007 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" event={"ID":"36cf5632-ebba-49b5-ad2c-40f5004570bd","Type":"ContainerStarted","Data":"b3cb0148bfe60edc2197e0272a90c6cc546a8ddb5a7ad6ac7c80fbef5bf53cd1"} Nov 02 22:21:36.970109 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:36.970056 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" event={"ID":"36cf5632-ebba-49b5-ad2c-40f5004570bd","Type":"ContainerStarted","Data":"29be60302f21def038fedf9c1347ce7f7fad555191f12f1883f4ac734a6df0d2"} Nov 02 22:21:37.081149 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:37.081089 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podStartSLOduration=2.081074323 podStartE2EDuration="2.081074323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:37.078613541 +0000 UTC m=+919.890943717" watchObservedRunningTime="2026-11-02 22:21:37.081074323 +0000 UTC m=+919.893404815" Nov 02 22:21:37.312847 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:37.312770 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:38.472228 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:38.472171 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:39.708203 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.708159 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz"] Nov 02 22:21:39.712369 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.712341 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.761506 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.761473 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-ca85db-openshift-default-dockercfg-bzw5r\"" Nov 02 22:21:39.810639 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.810602 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz"] Nov 02 22:21:39.859301 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.859246 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.859301 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.859302 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.859540 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.859422 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.859540 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.859474 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.859620 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.859557 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjk2h\" (UniqueName: \"kubernetes.io/projected/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-kube-api-access-wjk2h\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.859620 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.859591 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.859711 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.859632 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.859762 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.859742 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.859798 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.859771 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.961364 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.961260 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.961364 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.961338 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.961614 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.961404 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjk2h\" (UniqueName: \"kubernetes.io/projected/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-kube-api-access-wjk2h\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.961614 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.961449 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.961614 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.961507 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.961614 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.961543 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.961614 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.961569 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.962019 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.961650 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.962019 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.961680 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.962019 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.961743 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.962019 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.961933 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.962156 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.962065 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.962293 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.962255 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.962413 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.962301 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.964087 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.964058 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:39.964327 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:39.964309 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:40.019335 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:40.019298 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:40.019489 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:40.019398 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjk2h\" (UniqueName: \"kubernetes.io/projected/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-kube-api-access-wjk2h\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:40.026437 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:40.026411 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:40.461433 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:40.461393 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz"] Nov 02 22:21:40.467552 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:21:40.467518 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4824cb6c_15d6_4c48_8824_d7bf7b7932f1.slice/crio-abbb1f984cd20e5f7f1fa0d6f92df4a98744d9a3d5e61646888c264561d21a58 WatchSource:0}: Error finding container abbb1f984cd20e5f7f1fa0d6f92df4a98744d9a3d5e61646888c264561d21a58: Status 404 returned error can't find the container with id abbb1f984cd20e5f7f1fa0d6f92df4a98744d9a3d5e61646888c264561d21a58 Nov 02 22:21:40.469816 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:40.469782 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:21:40.469899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:40.469856 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:21:40.469946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:40.469905 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:21:40.996375 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:40.996330 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" event={"ID":"4824cb6c-15d6-4c48-8824-d7bf7b7932f1","Type":"ContainerStarted","Data":"6b3a2646fdbe2076bb588f5a9ce673950c8be725caea1fee22d5dc013b224c28"} Nov 02 22:21:40.996938 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:40.996408 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" event={"ID":"4824cb6c-15d6-4c48-8824-d7bf7b7932f1","Type":"ContainerStarted","Data":"abbb1f984cd20e5f7f1fa0d6f92df4a98744d9a3d5e61646888c264561d21a58"} Nov 02 22:21:41.027300 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:41.027250 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:41.029429 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:41.029395 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 22:21:41.029628 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:41.029456 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" podUID="4824cb6c-15d6-4c48-8824-d7bf7b7932f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 22:21:41.109925 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:41.109843 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" podStartSLOduration=2.109825413 podStartE2EDuration="2.109825413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:41.109030364 +0000 UTC m=+923.921360545" watchObservedRunningTime="2026-11-02 22:21:41.109825413 +0000 UTC m=+923.922155593" Nov 02 22:21:41.647075 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:41.647040 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:42.028061 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.027830 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 22:21:42.028061 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.027930 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" podUID="4824cb6c-15d6-4c48-8824-d7bf7b7932f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 22:21:42.566915 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.562166 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9"] Nov 02 22:21:42.582814 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.574568 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.615754 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.615717 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk"] Nov 02 22:21:42.649134 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.649104 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.689898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.686152 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9"] Nov 02 22:21:42.742924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.729723 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.742924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.729784 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.742924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.729837 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.742924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.729890 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.742924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.729919 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2jqm\" (UniqueName: \"kubernetes.io/projected/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-kube-api-access-w2jqm\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.742924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.729959 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.742924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.730015 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.742924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.730109 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.742924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.730160 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.742924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.741778 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk"] Nov 02 22:21:42.843697 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.843590 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.844032 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.844003 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.844186 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.844168 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.844303 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.844288 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.844457 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.844441 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.844620 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.844586 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.844753 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.844738 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.844901 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.844868 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.845134 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.845112 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.845329 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.845308 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.845484 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.845448 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.845609 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.845592 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjxm4\" (UniqueName: \"kubernetes.io/projected/57f82336-d76e-4f2b-b8e7-cb248650a5aa-kube-api-access-wjxm4\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.845767 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.845746 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.845905 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.845888 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.846045 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.846031 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.846169 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.846154 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jqm\" (UniqueName: \"kubernetes.io/projected/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-kube-api-access-w2jqm\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.846297 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.846271 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.846437 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.846421 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.847094 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.847073 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.848032 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.848012 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.848504 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.848481 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.848991 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.848959 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.863913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.852997 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.863913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.853619 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.895936 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.884551 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.936945 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.932707 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.958909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.948327 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.958909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.948376 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.958909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.948401 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.958909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.948476 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.958909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.948540 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.958909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.948661 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.958909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.948694 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjxm4\" (UniqueName: \"kubernetes.io/projected/57f82336-d76e-4f2b-b8e7-cb248650a5aa-kube-api-access-wjxm4\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.958909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.948736 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.958909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.948782 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.958909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.949323 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.958909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.949574 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.958909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.950139 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.958909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.951176 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.958909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.951694 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.992758 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.991524 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2jqm\" (UniqueName: \"kubernetes.io/projected/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-kube-api-access-w2jqm\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4j9\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:42.993423 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.993257 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:42.996336 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:42.996265 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:43.027164 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:43.027123 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjxm4\" (UniqueName: \"kubernetes.io/projected/57f82336-d76e-4f2b-b8e7-cb248650a5aa-kube-api-access-wjxm4\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:43.029095 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:43.029058 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 22:21:43.029556 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:43.029123 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" podUID="4824cb6c-15d6-4c48-8824-d7bf7b7932f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 22:21:43.031819 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:43.031759 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-f7qmk\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:43.097961 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:43.079063 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:43.241117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:43.240837 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:43.598723 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:43.598037 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk"] Nov 02 22:21:43.602026 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:21:43.601991 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f82336_d76e_4f2b_b8e7_cb248650a5aa.slice/crio-f300e89f1dc4bd8a8636be984824d9882091c2ce99cedc2dd088c7755016b4ae WatchSource:0}: Error finding container f300e89f1dc4bd8a8636be984824d9882091c2ce99cedc2dd088c7755016b4ae: Status 404 returned error can't find the container with id f300e89f1dc4bd8a8636be984824d9882091c2ce99cedc2dd088c7755016b4ae Nov 02 22:21:43.604839 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:43.604805 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:21:43.604965 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:43.604903 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:21:43.604965 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:43.604934 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:21:43.670846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:43.670768 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:43.716055 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:43.715713 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9"] Nov 02 22:21:43.723439 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:21:43.723403 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76eb16d5_a9c1_4fe6_a3ef_0f195f29e989.slice/crio-9cdf7edd970d22dd41d1fd1c7bfdfff592151d623f9625971621b8417d322e87 WatchSource:0}: Error finding container 9cdf7edd970d22dd41d1fd1c7bfdfff592151d623f9625971621b8417d322e87: Status 404 returned error can't find the container with id 9cdf7edd970d22dd41d1fd1c7bfdfff592151d623f9625971621b8417d322e87 Nov 02 22:21:43.727056 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:43.727023 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:21:43.727182 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:43.727092 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:21:43.727182 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:43.727145 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:21:43.822914 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:21:43.822809 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:21:44.027271 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:44.027172 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 22:21:44.027271 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:44.027227 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" podUID="4824cb6c-15d6-4c48-8824-d7bf7b7932f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 22:21:44.047632 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:44.047593 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" event={"ID":"57f82336-d76e-4f2b-b8e7-cb248650a5aa","Type":"ContainerStarted","Data":"365837f871487a8016a9a0eb4b623ca7649512ba38651880712bb8502260e6c8"} Nov 02 22:21:44.048158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:44.047658 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" event={"ID":"57f82336-d76e-4f2b-b8e7-cb248650a5aa","Type":"ContainerStarted","Data":"f300e89f1dc4bd8a8636be984824d9882091c2ce99cedc2dd088c7755016b4ae"} Nov 02 22:21:44.049941 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:44.049905 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" event={"ID":"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989","Type":"ContainerStarted","Data":"1fbf7ac78b8c975d7e4c2a88dc91db3d9aa12f27743bb7a8092a95b4a742a507"} Nov 02 22:21:44.050080 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:44.049968 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" event={"ID":"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989","Type":"ContainerStarted","Data":"9cdf7edd970d22dd41d1fd1c7bfdfff592151d623f9625971621b8417d322e87"} Nov 02 22:21:44.099617 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:44.099571 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:44.101757 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:44.101728 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f7qmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 22:21:44.101933 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:44.101785 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" podUID="57f82336-d76e-4f2b-b8e7-cb248650a5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 22:21:44.175913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:44.173322 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" podStartSLOduration=2.173301132 podStartE2EDuration="2.173301132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:44.171694764 +0000 UTC m=+926.984024942" watchObservedRunningTime="2026-11-02 22:21:44.173301132 +0000 UTC m=+926.985631305" Nov 02 22:21:44.241719 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:44.241295 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:44.243187 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:44.243157 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4t4j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 22:21:44.243326 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:44.243223 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" podUID="76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 22:21:44.286558 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:44.286430 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" podStartSLOduration=2.286411217 podStartE2EDuration="2.286411217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:44.284567418 +0000 UTC m=+927.096897595" watchObservedRunningTime="2026-11-02 22:21:44.286411217 +0000 UTC m=+927.098741400" Nov 02 22:21:45.027902 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:45.027351 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 22:21:45.027902 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:45.027431 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" podUID="4824cb6c-15d6-4c48-8824-d7bf7b7932f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 22:21:45.108898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:45.107017 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f7qmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 22:21:45.108898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:45.107081 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" podUID="57f82336-d76e-4f2b-b8e7-cb248650a5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 22:21:45.241627 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:45.241587 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4t4j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 22:21:45.241804 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:45.241642 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" podUID="76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 22:21:46.027897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:46.027366 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 22:21:46.027897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:46.027445 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" podUID="4824cb6c-15d6-4c48-8824-d7bf7b7932f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 22:21:46.112900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:46.112336 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f7qmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 22:21:46.112900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:46.112398 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" podUID="57f82336-d76e-4f2b-b8e7-cb248650a5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 22:21:46.241398 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:46.241356 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4t4j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 22:21:46.241573 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:46.241417 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" podUID="76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 22:21:46.336694 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:46.335978 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:46.381901 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:46.354306 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:47.027673 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:47.027633 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 22:21:47.027900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:47.027696 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" podUID="4824cb6c-15d6-4c48-8824-d7bf7b7932f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 22:21:47.111362 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:47.111320 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f7qmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 22:21:47.111672 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:47.111642 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" podUID="57f82336-d76e-4f2b-b8e7-cb248650a5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 22:21:47.148729 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:47.148698 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:21:47.241713 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:47.241679 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4t4j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 22:21:47.241977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:47.241953 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" podUID="76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 22:21:48.031609 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:48.031576 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:48.032203 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:48.032183 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:48.032772 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:48.032753 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:21:48.103336 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:48.103289 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f7qmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 22:21:48.103519 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:48.103353 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" podUID="57f82336-d76e-4f2b-b8e7-cb248650a5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 22:21:48.241506 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:48.241468 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4t4j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 22:21:48.241919 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:48.241529 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" podUID="76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 22:21:48.897901 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:48.887914 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:49.101902 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:49.101293 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f7qmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 22:21:49.101902 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:49.101362 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" podUID="57f82336-d76e-4f2b-b8e7-cb248650a5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 22:21:49.241451 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:49.241357 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4t4j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 22:21:49.241451 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:49.241415 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" podUID="76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 22:21:50.099563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:50.099518 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f7qmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 22:21:50.099785 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:50.099581 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" podUID="57f82336-d76e-4f2b-b8e7-cb248650a5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 22:21:50.241433 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:50.241395 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4t4j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 22:21:50.241650 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:50.241455 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" podUID="76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 22:21:51.099396 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:51.099355 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f7qmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 22:21:51.099616 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:51.099425 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" podUID="57f82336-d76e-4f2b-b8e7-cb248650a5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 22:21:51.241777 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:51.241641 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4t4j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 22:21:51.241777 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:51.241712 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" podUID="76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 22:21:52.588085 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:52.588052 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:52.590603 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:52.590575 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:52.591715 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:52.591689 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:21:52.652603 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:52.652568 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:52.653685 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:52.653662 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:52.654613 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:52.654593 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:21:53.779498 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.779465 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:53.828201 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.828166 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp"] Nov 02 22:21:53.833117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.833094 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.864486 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.864448 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.864704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.864554 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.864704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.864624 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.864796 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.864719 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.864796 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.864744 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.864796 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.864767 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.864930 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.864906 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.864970 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.864955 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9384cb9b-0f07-485e-9635-c7dc585a8b76-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.865052 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.865032 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8628\" (UniqueName: \"kubernetes.io/projected/9384cb9b-0f07-485e-9635-c7dc585a8b76-kube-api-access-p8628\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.881996 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.881968 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-fe810f-openshift-default-dockercfg-lk5vp\"" Nov 02 22:21:53.931308 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.931271 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp"] Nov 02 22:21:53.966218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.966171 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.966430 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.966227 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.966430 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.966366 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.966560 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.966434 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.966560 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.966496 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9384cb9b-0f07-485e-9635-c7dc585a8b76-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.966560 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.966553 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8628\" (UniqueName: \"kubernetes.io/projected/9384cb9b-0f07-485e-9635-c7dc585a8b76-kube-api-access-p8628\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.966736 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.966600 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.966736 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.966660 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.966831 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.966749 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.966992 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.966967 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.967346 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.967322 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.967506 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.967377 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.967580 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.967402 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.967580 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.967445 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9384cb9b-0f07-485e-9635-c7dc585a8b76-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.969079 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.969057 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:53.969214 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:53.969196 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:54.024381 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:54.024347 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:54.024518 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:54.024418 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8628\" (UniqueName: \"kubernetes.io/projected/9384cb9b-0f07-485e-9635-c7dc585a8b76-kube-api-access-p8628\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:54.147690 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:54.147655 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:54.581951 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:54.581855 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp"] Nov 02 22:21:54.587997 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:21:54.587960 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9384cb9b_0f07_485e_9635_c7dc585a8b76.slice/crio-164e8159b18d8ed2789ca3ae3bd3580e94ed010434a2d2095301ee497503b554 WatchSource:0}: Error finding container 164e8159b18d8ed2789ca3ae3bd3580e94ed010434a2d2095301ee497503b554: Status 404 returned error can't find the container with id 164e8159b18d8ed2789ca3ae3bd3580e94ed010434a2d2095301ee497503b554 Nov 02 22:21:54.590943 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:54.590902 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:21:54.591073 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:54.590977 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:21:54.591073 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:54.591010 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:21:55.207776 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:55.207740 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" event={"ID":"9384cb9b-0f07-485e-9635-c7dc585a8b76","Type":"ContainerStarted","Data":"4be053a3f19aa0ebcf22c1068d31aecb03b21ab04ef3181b992b2c0b7156ec5c"} Nov 02 22:21:55.208207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:55.207802 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" event={"ID":"9384cb9b-0f07-485e-9635-c7dc585a8b76","Type":"ContainerStarted","Data":"164e8159b18d8ed2789ca3ae3bd3580e94ed010434a2d2095301ee497503b554"} Nov 02 22:21:55.325821 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:55.325753 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" podStartSLOduration=2.325736337 podStartE2EDuration="2.325736337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:55.325408135 +0000 UTC m=+938.137738312" watchObservedRunningTime="2026-11-02 22:21:55.325736337 +0000 UTC m=+938.138066517" Nov 02 22:21:56.148859 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:56.148817 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:21:56.150567 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:56.150544 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 22:21:56.150660 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:56.150604 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" podUID="9384cb9b-0f07-485e-9635-c7dc585a8b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 22:21:57.148626 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:57.148585 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 22:21:57.149088 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:57.148639 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" podUID="9384cb9b-0f07-485e-9635-c7dc585a8b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 22:21:58.148555 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:58.148518 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 22:21:58.148773 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:58.148580 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" podUID="9384cb9b-0f07-485e-9635-c7dc585a8b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 22:21:58.638999 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:58.638959 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:21:59.148857 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:59.148820 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 22:21:59.149295 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:21:59.148890 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" podUID="9384cb9b-0f07-485e-9635-c7dc585a8b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 22:22:00.148946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:00.148913 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 22:22:00.149459 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:00.148984 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" podUID="9384cb9b-0f07-485e-9635-c7dc585a8b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 22:22:01.148133 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:01.148081 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 22:22:01.148297 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:01.148146 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" podUID="9384cb9b-0f07-485e-9635-c7dc585a8b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 22:22:01.393479 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:01.393440 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:22:02.560470 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:02.560432 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:02.706721 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:02.706689 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:22:02.707157 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:02.707130 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:22:02.708568 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:02.708546 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:22:03.681805 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:03.681773 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:08.608641 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:08.608591 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:13.393307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:13.393265 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:13.770464 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:22:13.770376 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:22:18.284186 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:18.284147 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:23.196834 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:23.196800 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:23.377691 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:23.377661 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:25.917024 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:25.916985 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:28.099222 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:28.099188 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:28.967962 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:28.967922 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz"] Nov 02 22:22:28.973204 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:28.973178 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.013109 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.013072 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-tmp-dir\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.013327 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.013268 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d76402c-4d39-4c03-8018-16f2b97b5fab-tls-certs\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.013407 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.013332 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-dshm\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.013470 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.013402 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmk8w\" (UniqueName: \"kubernetes.io/projected/5d76402c-4d39-4c03-8018-16f2b97b5fab-kube-api-access-pmk8w\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.013470 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.013445 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-model-cache\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.013578 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.013564 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-home\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.036095 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.036061 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz"] Nov 02 22:22:29.070820 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.070781 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8263a4\"/\"distinct-w2-8263a4-kserve-self-signed-certs\"" Nov 02 22:22:29.115699 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.115655 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-home\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.117007 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.115741 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-tmp-dir\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.117007 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.115869 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d76402c-4d39-4c03-8018-16f2b97b5fab-tls-certs\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.117007 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.115968 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-dshm\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.117007 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.116042 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmk8w\" (UniqueName: \"kubernetes.io/projected/5d76402c-4d39-4c03-8018-16f2b97b5fab-kube-api-access-pmk8w\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.117007 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.116087 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-model-cache\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.117007 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.116177 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-home\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.117007 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.116243 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-tmp-dir\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.117007 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.116790 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-model-cache\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.119331 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.119306 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d76402c-4d39-4c03-8018-16f2b97b5fab-tls-certs\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.129033 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.128993 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-dshm\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.152320 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.152275 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:29.174392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.174361 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmk8w\" (UniqueName: \"kubernetes.io/projected/5d76402c-4d39-4c03-8018-16f2b97b5fab-kube-api-access-pmk8w\") pod \"distinct-w2-8263a4-kserve-94947676c-fblwz\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.286145 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.286039 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:29.680438 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:29.680400 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz"] Nov 02 22:22:29.687062 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:22:29.687030 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d76402c_4d39_4c03_8018_16f2b97b5fab.slice/crio-393ca4b6ae2a0accc4c04997036165e0bac0e2edb9cc52f24fee44d41d2da780 WatchSource:0}: Error finding container 393ca4b6ae2a0accc4c04997036165e0bac0e2edb9cc52f24fee44d41d2da780: Status 404 returned error can't find the container with id 393ca4b6ae2a0accc4c04997036165e0bac0e2edb9cc52f24fee44d41d2da780 Nov 02 22:22:30.434680 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:30.434640 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:30.450890 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:30.450837 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" event={"ID":"5d76402c-4d39-4c03-8018-16f2b97b5fab","Type":"ContainerStarted","Data":"c5859f1b20c931e286714f4815b82f7b86b1665c5cbd617df273cb8b5264c760"} Nov 02 22:22:30.451086 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:30.450902 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" event={"ID":"5d76402c-4d39-4c03-8018-16f2b97b5fab","Type":"ContainerStarted","Data":"393ca4b6ae2a0accc4c04997036165e0bac0e2edb9cc52f24fee44d41d2da780"} Nov 02 22:22:30.572084 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:30.572022 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podStartSLOduration=2.572007727 podStartE2EDuration="2.572007727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:30.569288458 +0000 UTC m=+973.381618640" watchObservedRunningTime="2026-11-02 22:22:30.572007727 +0000 UTC m=+973.384337904" Nov 02 22:22:30.618438 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:30.618404 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:31.286976 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:31.286936 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:32.976644 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:32.976593 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:33.998109 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:33.998070 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:35.937596 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:35.937558 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:22:38.079774 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:38.079740 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:39.287075 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:39.287033 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:39.299721 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:39.299688 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:39.523146 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:39.523113 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:22:41.992743 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:41.992707 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:42.880290 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:42.880249 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:43.758158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:43.758117 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:43.781011 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:22:43.780950 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:22:47.685803 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:47.685771 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:52.491610 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:52.491568 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:54.000477 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.000438 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p"] Nov 02 22:22:54.004692 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.004670 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.078724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.078690 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p"] Nov 02 22:22:54.124676 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.124629 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.124863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.124710 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.124863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.124829 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.124863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.124860 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.125083 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.125006 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.125083 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.125061 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.125200 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.125166 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qdz7\" (UniqueName: \"kubernetes.io/projected/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-kube-api-access-6qdz7\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.125258 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.125213 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.125314 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.125256 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.226289 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.226248 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.226506 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.226330 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.226506 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.226349 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.226506 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.226382 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.226506 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.226414 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.226506 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.226485 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qdz7\" (UniqueName: \"kubernetes.io/projected/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-kube-api-access-6qdz7\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.226782 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.226520 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.226782 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.226560 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.226782 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.226641 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.227067 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.227039 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.227199 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.227113 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.227199 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.227135 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.227312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.227292 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.227348 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.227335 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.228666 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.228642 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.229344 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.229320 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.284223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.284149 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.284376 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.284353 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qdz7\" (UniqueName: \"kubernetes.io/projected/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-kube-api-access-6qdz7\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.319337 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.319301 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:54.750637 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.750602 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p"] Nov 02 22:22:54.757695 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:22:54.757663 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30a7ffb_0fbd_437c_af03_a89ed7449bd4.slice/crio-c60a9031536d5ed16033f0ffd5152f1cd6ffe2cc0f7956f8d4da22554f7ab927 WatchSource:0}: Error finding container c60a9031536d5ed16033f0ffd5152f1cd6ffe2cc0f7956f8d4da22554f7ab927: Status 404 returned error can't find the container with id c60a9031536d5ed16033f0ffd5152f1cd6ffe2cc0f7956f8d4da22554f7ab927 Nov 02 22:22:54.759909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.759865 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:22:54.759986 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.759948 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:22:54.760025 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:54.759983 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:22:55.619362 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:55.619324 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" event={"ID":"c30a7ffb-0fbd-437c-af03-a89ed7449bd4","Type":"ContainerStarted","Data":"6972d6132a0eed01e629f50a65098a05fc1f90068a427ff2e043fe85b199a2c3"} Nov 02 22:22:55.619867 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:55.619384 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" event={"ID":"c30a7ffb-0fbd-437c-af03-a89ed7449bd4","Type":"ContainerStarted","Data":"c60a9031536d5ed16033f0ffd5152f1cd6ffe2cc0f7956f8d4da22554f7ab927"} Nov 02 22:22:55.786471 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:55.786417 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" podStartSLOduration=2.786397762 podStartE2EDuration="2.786397762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:55.784823147 +0000 UTC m=+998.597153350" watchObservedRunningTime="2026-11-02 22:22:55.786397762 +0000 UTC m=+998.598728021" Nov 02 22:22:56.319884 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:56.319831 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:22:56.321608 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:56.321582 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 22:22:56.321740 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:56.321633 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" podUID="c30a7ffb-0fbd-437c-af03-a89ed7449bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 22:22:57.315257 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:57.315189 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:22:57.320326 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:57.320298 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 22:22:57.320484 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:57.320355 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" podUID="c30a7ffb-0fbd-437c-af03-a89ed7449bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 22:22:58.320071 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:58.320042 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 22:22:58.320513 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:58.320098 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" podUID="c30a7ffb-0fbd-437c-af03-a89ed7449bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 22:22:59.324007 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:59.323971 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 22:22:59.324659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:22:59.324610 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" podUID="c30a7ffb-0fbd-437c-af03-a89ed7449bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 22:23:00.320516 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:00.320472 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 22:23:00.320706 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:00.320530 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" podUID="c30a7ffb-0fbd-437c-af03-a89ed7449bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 22:23:01.319942 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:01.319908 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 22:23:01.320318 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:01.319960 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" podUID="c30a7ffb-0fbd-437c-af03-a89ed7449bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 22:23:02.324935 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:02.324904 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:23:02.325362 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:02.325294 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:23:02.325974 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:02.325957 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:23:02.367155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:02.367121 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:02.900625 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:02.900590 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:03.118602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:03.118553 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:07.383187 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:07.383150 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:12.184142 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:12.184105 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:12.363071 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:12.363033 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:23:13.777529 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:23:13.777459 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:23:14.164117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.164080 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr"] Nov 02 22:23:14.168627 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.168603 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.224797 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.224760 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr"] Nov 02 22:23:14.225456 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.225423 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c422c51d-7d39-4751-92c5-0de8ca3880a0-tls-certs\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.225656 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.225633 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-home\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.225778 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.225682 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-model-cache\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.225887 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.225845 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-dshm\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.226083 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.226060 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55bfp\" (UniqueName: \"kubernetes.io/projected/c422c51d-7d39-4751-92c5-0de8ca3880a0-kube-api-access-55bfp\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.226179 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.226095 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-tmp-dir\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.266568 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.266533 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-711bd4\"/\"distinct-w0-711bd4-kserve-self-signed-certs\"" Nov 02 22:23:14.266568 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.266532 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-711bd4\"/\"openshift-service-ca.crt\"" Nov 02 22:23:14.266790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.266610 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-711bd4\"/\"default-dockercfg-46b9k\"" Nov 02 22:23:14.266790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.266614 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-711bd4\"/\"kube-root-ca.crt\"" Nov 02 22:23:14.298141 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.298107 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:14.329773 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.329728 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-tmp-dir\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.330026 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.329993 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c422c51d-7d39-4751-92c5-0de8ca3880a0-tls-certs\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.330104 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.330029 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-home\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.330104 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.330069 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-model-cache\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.330216 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.330188 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-dshm\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.330304 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.330288 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55bfp\" (UniqueName: \"kubernetes.io/projected/c422c51d-7d39-4751-92c5-0de8ca3880a0-kube-api-access-55bfp\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.330740 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.330450 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-tmp-dir\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.331406 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.331380 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-model-cache\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.331621 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.331588 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-home\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.334436 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.334413 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c422c51d-7d39-4751-92c5-0de8ca3880a0-tls-certs\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.334992 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.334963 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-dshm\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.388724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.388685 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55bfp\" (UniqueName: \"kubernetes.io/projected/c422c51d-7d39-4751-92c5-0de8ca3880a0-kube-api-access-55bfp\") pod \"distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.481752 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.481659 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:14.860135 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:14.860049 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr"] Nov 02 22:23:14.865955 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:23:14.865927 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc422c51d_7d39_4751_92c5_0de8ca3880a0.slice/crio-ba93be5c11dadd7c02b888b94757ce0f074de32557f72c5775c9b6be943b858b WatchSource:0}: Error finding container ba93be5c11dadd7c02b888b94757ce0f074de32557f72c5775c9b6be943b858b: Status 404 returned error can't find the container with id ba93be5c11dadd7c02b888b94757ce0f074de32557f72c5775c9b6be943b858b Nov 02 22:23:15.769546 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:15.769502 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" event={"ID":"c422c51d-7d39-4751-92c5-0de8ca3880a0","Type":"ContainerStarted","Data":"56c4dabe19fcafc8494fb96e1a0aded7347241adcb50973455eef756f7f83129"} Nov 02 22:23:15.769546 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:15.769553 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" event={"ID":"c422c51d-7d39-4751-92c5-0de8ca3880a0","Type":"ContainerStarted","Data":"ba93be5c11dadd7c02b888b94757ce0f074de32557f72c5775c9b6be943b858b"} Nov 02 22:23:15.881219 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:15.881158 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podStartSLOduration=1.881135743 podStartE2EDuration="1.881135743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:23:15.879030529 +0000 UTC m=+1018.691360704" watchObservedRunningTime="2026-11-02 22:23:15.881135743 +0000 UTC m=+1018.693465921" Nov 02 22:23:16.481987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:16.481949 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:17.092191 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:17.092157 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:21.897064 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:21.897028 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:24.482781 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:24.482745 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:24.495834 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:24.495801 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:24.845042 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:24.844966 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:23:26.694490 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:26.694457 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:31.592122 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:31.592077 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:33.664561 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:33.664521 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:36.394593 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:36.394556 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:41.282959 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:41.282918 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:42.077764 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.077725 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq"] Nov 02 22:23:42.083465 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.083438 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.138172 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.138129 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq"] Nov 02 22:23:42.179407 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.179373 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-98a222\"/\"distinct-2-w3-98a222-kserve-self-signed-certs\"" Nov 02 22:23:42.196820 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.196780 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-tls-certs\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.197060 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.196975 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-home\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.197060 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.197014 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-dshm\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.197145 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.197084 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-tmp-dir\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.197145 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.197139 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w27w2\" (UniqueName: \"kubernetes.io/projected/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-kube-api-access-w27w2\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.197233 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.197215 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-model-cache\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.218560 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.218527 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:42.298532 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.298500 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-tmp-dir\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.298985 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.298570 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w27w2\" (UniqueName: \"kubernetes.io/projected/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-kube-api-access-w27w2\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.298985 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.298595 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-model-cache\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.298985 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.298649 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-tls-certs\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.298985 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.298776 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-home\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.298985 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.298797 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-dshm\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.299201 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.298999 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-tmp-dir\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.299201 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.299073 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-model-cache\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.299201 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.299141 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-home\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.301063 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.301042 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-dshm\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.301219 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.301201 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-tls-certs\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.356792 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.356713 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w27w2\" (UniqueName: \"kubernetes.io/projected/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-kube-api-access-w27w2\") pod \"distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.393315 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.393280 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:42.395274 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.395252 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:42.778656 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.778619 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq"] Nov 02 22:23:42.784664 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:23:42.784635 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c57bc80_ff20_4fc4_a19d_5dce6dc11ede.slice/crio-1284a6439dfb4dbc834fd9328e6cd58d91b9924f54fa0474a2bbaa3e117b1c12 WatchSource:0}: Error finding container 1284a6439dfb4dbc834fd9328e6cd58d91b9924f54fa0474a2bbaa3e117b1c12: Status 404 returned error can't find the container with id 1284a6439dfb4dbc834fd9328e6cd58d91b9924f54fa0474a2bbaa3e117b1c12 Nov 02 22:23:42.964863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.964828 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" event={"ID":"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede","Type":"ContainerStarted","Data":"42c41e097e7777554b7eaf25f45e1a05d94685c760e342684658c3cd0baccb5c"} Nov 02 22:23:42.965165 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:42.964889 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" event={"ID":"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede","Type":"ContainerStarted","Data":"1284a6439dfb4dbc834fd9328e6cd58d91b9924f54fa0474a2bbaa3e117b1c12"} Nov 02 22:23:43.075955 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:43.075815 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podStartSLOduration=1.075779978 podStartE2EDuration="1.075779978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:23:43.074935979 +0000 UTC m=+1045.887266158" watchObservedRunningTime="2026-11-02 22:23:43.075779978 +0000 UTC m=+1045.888110159" Nov 02 22:23:43.395705 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:43.395669 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:43.770442 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:23:43.770356 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:23:46.302684 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:46.302645 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:51.164659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:51.164618 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:52.395450 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:52.395410 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:52.408456 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:52.408421 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:52.673433 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:52.673335 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:23:53.050072 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:53.049989 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:23:54.104918 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.104883 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z"] Nov 02 22:23:54.109475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.109448 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.182423 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.182375 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8e2845-337e-4fa5-b745-ae219273156b-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.182423 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.182433 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8e2845-337e-4fa5-b745-ae219273156b-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.182714 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.182591 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6tmx\" (UniqueName: \"kubernetes.io/projected/9b8e2845-337e-4fa5-b745-ae219273156b-kube-api-access-t6tmx\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.182714 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.182647 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.182838 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.182770 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.182946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.182847 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.182946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.182889 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.183079 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.182956 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.183079 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.182992 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8e2845-337e-4fa5-b745-ae219273156b-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.210344 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.210300 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z"] Nov 02 22:23:54.284107 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.284069 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6tmx\" (UniqueName: \"kubernetes.io/projected/9b8e2845-337e-4fa5-b745-ae219273156b-kube-api-access-t6tmx\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.284350 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.284121 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.284350 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.284165 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.284350 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.284221 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.284350 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.284246 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.284350 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.284308 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.284350 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.284341 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8e2845-337e-4fa5-b745-ae219273156b-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.284705 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.284466 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8e2845-337e-4fa5-b745-ae219273156b-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.284705 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.284496 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8e2845-337e-4fa5-b745-ae219273156b-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.284705 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.284600 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.284705 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.284667 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.284977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.284748 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.285161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.285124 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.285438 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.285414 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8e2845-337e-4fa5-b745-ae219273156b-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.287086 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.287062 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.287235 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.287215 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8e2845-337e-4fa5-b745-ae219273156b-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.341618 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.341577 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8e2845-337e-4fa5-b745-ae219273156b-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.341794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.341753 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6tmx\" (UniqueName: \"kubernetes.io/projected/9b8e2845-337e-4fa5-b745-ae219273156b-kube-api-access-t6tmx\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.428455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.428415 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:54.866846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.866802 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z"] Nov 02 22:23:54.872206 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:23:54.872169 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8e2845_337e_4fa5_b745_ae219273156b.slice/crio-c15ca5bf4496ea74b52d73d19cb87b8bd3b1a11727c2ea1287399b6d02145a3c WatchSource:0}: Error finding container c15ca5bf4496ea74b52d73d19cb87b8bd3b1a11727c2ea1287399b6d02145a3c: Status 404 returned error can't find the container with id c15ca5bf4496ea74b52d73d19cb87b8bd3b1a11727c2ea1287399b6d02145a3c Nov 02 22:23:54.874748 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.874716 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:23:54.874834 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.874783 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:23:54.874834 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:54.874812 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:23:55.056097 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:55.056058 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" event={"ID":"9b8e2845-337e-4fa5-b745-ae219273156b","Type":"ContainerStarted","Data":"bf95e4eb6003ab689024a0bfebb1fbac9484c0c47ed01b17e21a079f5e5516cd"} Nov 02 22:23:55.056299 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:55.056125 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" event={"ID":"9b8e2845-337e-4fa5-b745-ae219273156b","Type":"ContainerStarted","Data":"c15ca5bf4496ea74b52d73d19cb87b8bd3b1a11727c2ea1287399b6d02145a3c"} Nov 02 22:23:55.171165 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:55.171039 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" podStartSLOduration=1.171019027 podStartE2EDuration="1.171019027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:23:55.169329797 +0000 UTC m=+1057.981659976" watchObservedRunningTime="2026-11-02 22:23:55.171019027 +0000 UTC m=+1057.983349206" Nov 02 22:23:55.428981 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:55.428856 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:23:55.430584 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:55.430560 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 22:23:55.430716 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:55.430607 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" podUID="9b8e2845-337e-4fa5-b745-ae219273156b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 22:23:56.112532 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:56.112489 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:23:56.429384 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:56.429292 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 22:23:56.429384 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:56.429357 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" podUID="9b8e2845-337e-4fa5-b745-ae219273156b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 22:23:57.429828 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:57.429780 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 22:23:57.430267 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:57.429845 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" podUID="9b8e2845-337e-4fa5-b745-ae219273156b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 22:23:58.429318 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:58.429277 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 22:23:58.429502 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:58.429339 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" podUID="9b8e2845-337e-4fa5-b745-ae219273156b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 22:23:59.429380 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:59.429344 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 22:23:59.430056 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:23:59.429418 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" podUID="9b8e2845-337e-4fa5-b745-ae219273156b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 22:24:00.429358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:00.429314 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 22:24:00.429551 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:00.429374 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" podUID="9b8e2845-337e-4fa5-b745-ae219273156b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 22:24:01.009907 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:01.009838 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:01.429385 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:01.429343 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 22:24:01.429614 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:01.429404 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" podUID="9b8e2845-337e-4fa5-b745-ae219273156b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 22:24:02.433705 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:02.433674 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:24:02.434221 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:02.434088 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:24:02.434714 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:02.434692 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:24:03.244000 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:03.243965 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:05.776595 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:05.776555 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:08.737920 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:08.737881 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:10.584577 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:10.584537 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:13.771428 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:24:13.771390 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:24:15.501728 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:15.501684 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:20.473638 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:20.473600 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:25.282023 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:25.281910 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:29.219182 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:29.219136 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:30.078992 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:30.078956 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:34.019520 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:34.019443 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:34.974595 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:34.974561 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:38.389252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:38.389207 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:38.572346 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:38.572313 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:39.132794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.132758 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk"] Nov 02 22:24:39.136869 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.136845 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.182799 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.182764 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.183005 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.182814 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.183005 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.182942 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.183005 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.182992 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.183125 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.183015 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.183184 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.183159 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.183219 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.183209 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.183294 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.183282 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.183328 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.183318 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkvg4\" (UniqueName: \"kubernetes.io/projected/8c608b09-9551-4d42-91c6-f6cb1a38bab7-kube-api-access-jkvg4\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.236837 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.236799 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk"] Nov 02 22:24:39.284211 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.284177 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.284428 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.284233 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkvg4\" (UniqueName: \"kubernetes.io/projected/8c608b09-9551-4d42-91c6-f6cb1a38bab7-kube-api-access-jkvg4\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.284428 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.284290 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.284428 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.284355 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.284428 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.284416 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.284643 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.284464 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.284643 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.284504 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.284747 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.284658 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.284747 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.284715 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.284850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.284775 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.284850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.284809 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.285010 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.284992 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.285149 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.285126 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.285413 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.285387 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.287085 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.287061 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.287276 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.287251 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.346591 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.346554 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.346775 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.346758 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkvg4\" (UniqueName: \"kubernetes.io/projected/8c608b09-9551-4d42-91c6-f6cb1a38bab7-kube-api-access-jkvg4\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.455336 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.455234 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:39.889426 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.889387 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk"] Nov 02 22:24:39.894696 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:24:39.894670 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c608b09_9551_4d42_91c6_f6cb1a38bab7.slice/crio-02bdbb9022c35f62188f6a05839d9b44096b16173e585e177c4bdb113e52fea2 WatchSource:0}: Error finding container 02bdbb9022c35f62188f6a05839d9b44096b16173e585e177c4bdb113e52fea2: Status 404 returned error can't find the container with id 02bdbb9022c35f62188f6a05839d9b44096b16173e585e177c4bdb113e52fea2 Nov 02 22:24:39.897057 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.897028 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:24:39.897128 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.897114 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:24:39.897171 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.897151 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:24:39.995909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:39.992847 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:40.388227 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:40.388185 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" event={"ID":"8c608b09-9551-4d42-91c6-f6cb1a38bab7","Type":"ContainerStarted","Data":"258d9d53311f4cf9fbd2fd81502d4e949e42a1eed833affbb62ed555fa88b532"} Nov 02 22:24:40.388420 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:40.388281 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" event={"ID":"8c608b09-9551-4d42-91c6-f6cb1a38bab7","Type":"ContainerStarted","Data":"02bdbb9022c35f62188f6a05839d9b44096b16173e585e177c4bdb113e52fea2"} Nov 02 22:24:40.455740 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:40.455702 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:40.457598 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:40.457572 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 22:24:40.457717 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:40.457651 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" podUID="8c608b09-9551-4d42-91c6-f6cb1a38bab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 22:24:40.505214 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:40.505162 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" podStartSLOduration=1.505139539 podStartE2EDuration="1.505139539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:24:40.501152082 +0000 UTC m=+1103.313482261" watchObservedRunningTime="2026-11-02 22:24:40.505139539 +0000 UTC m=+1103.317469716" Nov 02 22:24:41.476898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:41.466353 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 22:24:41.476898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:41.466436 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" podUID="8c608b09-9551-4d42-91c6-f6cb1a38bab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 22:24:42.447226 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:42.447179 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:24:42.477901 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:42.477389 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 22:24:42.477901 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:42.477448 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" podUID="8c608b09-9551-4d42-91c6-f6cb1a38bab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 22:24:42.731096 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:42.731014 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:43.148981 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:43.148943 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:43.457960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:43.457269 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 22:24:43.457960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:43.457346 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" podUID="8c608b09-9551-4d42-91c6-f6cb1a38bab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 22:24:43.791703 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:24:43.791614 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:24:44.475898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:44.475336 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 22:24:44.475898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:44.475410 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" podUID="8c608b09-9551-4d42-91c6-f6cb1a38bab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 22:24:44.917010 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:44.916966 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:45.474898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:45.474308 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 22:24:45.474898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:45.474376 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" podUID="8c608b09-9551-4d42-91c6-f6cb1a38bab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 22:24:46.455622 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:46.455576 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 22:24:46.456126 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:46.455642 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" podUID="8c608b09-9551-4d42-91c6-f6cb1a38bab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 22:24:47.573967 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:47.573073 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:47.574486 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:47.574024 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:47.575007 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:47.574982 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:24:49.781950 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:49.781911 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:51.487940 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:51.487893 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:54.682401 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:54.682366 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:56.622861 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.622819 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k"] Nov 02 22:24:56.628114 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.628095 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.678300 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.678231 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fe810f\"/\"openshift-service-ca.crt\"" Nov 02 22:24:56.724377 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.724337 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fe810f\"/\"kube-root-ca.crt\"" Nov 02 22:24:56.724563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.724384 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fe810f\"/\"default-dockercfg-svmfr\"" Nov 02 22:24:56.724563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.724535 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fe810f\"/\"distinct-2-w5-fe810f-kserve-self-signed-certs\"" Nov 02 22:24:56.728109 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.728087 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k"] Nov 02 22:24:56.742733 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.742702 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-home\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.742891 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.742734 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-tmp-dir\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.742891 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.742857 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-dshm\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.743053 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.743036 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-model-cache\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.743092 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.743063 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b70d1226-ba5d-4cf0-af76-4852ce82bd76-tls-certs\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.743179 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.743164 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf7kk\" (UniqueName: \"kubernetes.io/projected/b70d1226-ba5d-4cf0-af76-4852ce82bd76-kube-api-access-zf7kk\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.755941 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.755912 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:24:56.844455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.844417 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf7kk\" (UniqueName: \"kubernetes.io/projected/b70d1226-ba5d-4cf0-af76-4852ce82bd76-kube-api-access-zf7kk\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.844640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.844495 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-home\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.844640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.844531 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-tmp-dir\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.844640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.844609 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-dshm\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.844813 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.844749 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-model-cache\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.844813 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.844774 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b70d1226-ba5d-4cf0-af76-4852ce82bd76-tls-certs\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.844972 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.844951 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-home\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.845363 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.845336 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-tmp-dir\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.845500 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.845365 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-model-cache\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.847227 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.847201 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-dshm\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.847449 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.847431 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b70d1226-ba5d-4cf0-af76-4852ce82bd76-tls-certs\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.903070 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.902996 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf7kk\" (UniqueName: \"kubernetes.io/projected/b70d1226-ba5d-4cf0-af76-4852ce82bd76-kube-api-access-zf7kk\") pod \"distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:56.941985 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:56.941951 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:57.326088 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:57.326010 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k"] Nov 02 22:24:57.332100 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:24:57.332072 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb70d1226_ba5d_4cf0_af76_4852ce82bd76.slice/crio-ec7a9d9adaebd58cc002efca6afc6d12c3612f331bfef3948869f1a0e18604f1 WatchSource:0}: Error finding container ec7a9d9adaebd58cc002efca6afc6d12c3612f331bfef3948869f1a0e18604f1: Status 404 returned error can't find the container with id ec7a9d9adaebd58cc002efca6afc6d12c3612f331bfef3948869f1a0e18604f1 Nov 02 22:24:57.518132 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:57.518093 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" event={"ID":"b70d1226-ba5d-4cf0-af76-4852ce82bd76","Type":"ContainerStarted","Data":"2ee1e0497a1aa715e91b680a483f363e255ec8d54f2a2de15c8b3a5430a60452"} Nov 02 22:24:57.518132 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:57.518137 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" event={"ID":"b70d1226-ba5d-4cf0-af76-4852ce82bd76","Type":"ContainerStarted","Data":"ec7a9d9adaebd58cc002efca6afc6d12c3612f331bfef3948869f1a0e18604f1"} Nov 02 22:24:57.629979 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:57.629924 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podStartSLOduration=1.6299078219999998 podStartE2EDuration="1.629907822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:24:57.627332505 +0000 UTC m=+1120.439662683" watchObservedRunningTime="2026-11-02 22:24:57.629907822 +0000 UTC m=+1120.442238032" Nov 02 22:24:57.942473 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:57.942386 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:24:59.551354 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:24:59.551323 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:03.674898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:03.673847 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:04.866623 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:04.866577 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:06.942495 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:06.942456 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:25:06.955512 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:06.955485 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:25:07.626080 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:07.626042 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:25:09.672742 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:09.672708 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:12.049105 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.049070 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff"] Nov 02 22:25:12.053400 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.053379 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.149943 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.149912 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff"] Nov 02 22:25:12.150458 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.150438 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ca85db\"/\"distinct-2-w1-ca85db-kserve-self-signed-certs\"" Nov 02 22:25:12.171295 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.171262 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-tmp-dir\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.171475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.171309 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-model-cache\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.171544 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.171498 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-tls-certs\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.171603 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.171538 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2n7j\" (UniqueName: \"kubernetes.io/projected/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-kube-api-access-n2n7j\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.171717 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.171701 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-dshm\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.171763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.171729 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-home\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.236726 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.236692 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:12.272856 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.272820 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-dshm\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.272856 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.272863 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-home\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.273186 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.272949 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-tmp-dir\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.273186 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.272980 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-model-cache\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.273186 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.273093 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-tls-certs\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.273186 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.273119 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2n7j\" (UniqueName: \"kubernetes.io/projected/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-kube-api-access-n2n7j\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.273415 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.273397 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-home\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.273471 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.273421 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-tmp-dir\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.273471 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.273441 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-model-cache\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.275217 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.275191 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-dshm\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.275578 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.275551 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-tls-certs\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.330860 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.330786 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2n7j\" (UniqueName: \"kubernetes.io/projected/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-kube-api-access-n2n7j\") pod \"distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.365762 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.365731 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:12.784107 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:12.784074 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff"] Nov 02 22:25:12.789531 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:25:12.789506 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4c1aa58_b96e_4539_b445_62d8cb67ffa9.slice/crio-c8c6b5f97ef1c4317e2fdc56e21b749a79124fc3480cce1aa7d78966092f2389 WatchSource:0}: Error finding container c8c6b5f97ef1c4317e2fdc56e21b749a79124fc3480cce1aa7d78966092f2389: Status 404 returned error can't find the container with id c8c6b5f97ef1c4317e2fdc56e21b749a79124fc3480cce1aa7d78966092f2389 Nov 02 22:25:13.659933 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:13.659890 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" event={"ID":"b4c1aa58-b96e-4539-b445-62d8cb67ffa9","Type":"ContainerStarted","Data":"dd266b49e8354c4975453b18d78102174471c973a05c7cf985ab56a07348b01e"} Nov 02 22:25:13.660325 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:13.659948 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" event={"ID":"b4c1aa58-b96e-4539-b445-62d8cb67ffa9","Type":"ContainerStarted","Data":"c8c6b5f97ef1c4317e2fdc56e21b749a79124fc3480cce1aa7d78966092f2389"} Nov 02 22:25:13.771211 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:13.771156 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podStartSLOduration=1.771139145 podStartE2EDuration="1.771139145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:13.769293937 +0000 UTC m=+1136.581624116" watchObservedRunningTime="2026-11-02 22:25:13.771139145 +0000 UTC m=+1136.583469323" Nov 02 22:25:13.771406 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:25:13.771374 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:25:14.365927 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:14.365856 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:14.619451 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:14.619356 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:19.484077 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:19.484033 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:21.033988 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:21.033954 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:22.366049 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:22.366008 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:22.379505 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:22.379463 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:22.735231 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:22.735152 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:25:24.386706 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:24.386662 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:25.089826 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.089786 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c"] Nov 02 22:25:25.094326 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.094304 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.137089 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.137052 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.137273 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.137165 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c46dd84-f664-4317-86e5-035c682cd8da-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.137273 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.137208 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.137392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.137297 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.137392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.137343 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.137392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.137384 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6tjf\" (UniqueName: \"kubernetes.io/projected/1c46dd84-f664-4317-86e5-035c682cd8da-kube-api-access-l6tjf\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.137581 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.137441 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c46dd84-f664-4317-86e5-035c682cd8da-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.137581 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.137527 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c46dd84-f664-4317-86e5-035c682cd8da-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.137695 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.137594 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.238655 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.238616 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.238862 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.238782 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.238987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.238906 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c46dd84-f664-4317-86e5-035c682cd8da-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.238987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.238956 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.239100 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.239044 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-workload-certs\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.239100 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.239072 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.239214 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.239104 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.239214 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.239154 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-credential-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.239214 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.239161 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6tjf\" (UniqueName: \"kubernetes.io/projected/1c46dd84-f664-4317-86e5-035c682cd8da-kube-api-access-l6tjf\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.239370 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.239241 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c46dd84-f664-4317-86e5-035c682cd8da-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.239370 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.239314 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c46dd84-f664-4317-86e5-035c682cd8da-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.239469 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.239424 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-workload-socket\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.239563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.239540 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-istio-data\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.240184 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.240158 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c46dd84-f664-4317-86e5-035c682cd8da-istiod-ca-cert\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.241341 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.241316 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-istio-envoy\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.241848 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.241827 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c46dd84-f664-4317-86e5-035c682cd8da-istio-podinfo\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.296475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.296426 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c"] Nov 02 22:25:25.305387 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.305350 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6tjf\" (UniqueName: \"kubernetes.io/projected/1c46dd84-f664-4317-86e5-035c682cd8da-kube-api-access-l6tjf\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.306763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.306732 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c46dd84-f664-4317-86e5-035c682cd8da-istio-token\") pod \"e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.408180 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.408143 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:25.845557 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.845460 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c"] Nov 02 22:25:25.850863 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:25:25.850824 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c46dd84_f664_4317_86e5_035c682cd8da.slice/crio-204cc4a40b6a69fcfa09309661433c12fdaea3cb956f9486a15902484b0b69fb WatchSource:0}: Error finding container 204cc4a40b6a69fcfa09309661433c12fdaea3cb956f9486a15902484b0b69fb: Status 404 returned error can't find the container with id 204cc4a40b6a69fcfa09309661433c12fdaea3cb956f9486a15902484b0b69fb Nov 02 22:25:25.853468 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.853436 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:25:25.853549 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.853507 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:25:25.853549 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:25.853539 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:25:26.085505 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.085467 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6"] Nov 02 22:25:26.089784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.089757 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.150354 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.150317 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.150546 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.150372 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.150546 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.150400 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.150546 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.150401 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6"] Nov 02 22:25:26.150546 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.150450 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.150546 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.150512 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.150835 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.150674 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.150835 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.150717 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz9fh\" (UniqueName: \"kubernetes.io/projected/5c65b13b-f4dd-4a8a-9844-a0f0309be485-kube-api-access-kz9fh\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.150835 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.150771 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.150835 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.150814 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.252719 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.252686 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.252910 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.252732 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.252910 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.252764 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.252910 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.252814 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.252910 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.252849 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.253131 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.252916 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.253131 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.252948 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz9fh\" (UniqueName: \"kubernetes.io/projected/5c65b13b-f4dd-4a8a-9844-a0f0309be485-kube-api-access-kz9fh\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.253131 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.252976 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.253131 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.253016 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.253470 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.253448 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.253693 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.253664 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.253793 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.253770 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.253886 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.253853 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.253974 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.253954 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.255559 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.255540 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.255972 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.255951 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.311286 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.311246 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.311468 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.311361 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz9fh\" (UniqueName: \"kubernetes.io/projected/5c65b13b-f4dd-4a8a-9844-a0f0309be485-kube-api-access-kz9fh\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.405026 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.404990 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:26.756957 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.753110 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" event={"ID":"1c46dd84-f664-4317-86e5-035c682cd8da","Type":"ContainerStarted","Data":"bd19281ed8bbd6a5c820345f2d3db518bc5d2636b29fc4084904eca820ea5680"} Nov 02 22:25:26.756957 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.753191 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" event={"ID":"1c46dd84-f664-4317-86e5-035c682cd8da","Type":"ContainerStarted","Data":"204cc4a40b6a69fcfa09309661433c12fdaea3cb956f9486a15902484b0b69fb"} Nov 02 22:25:26.840376 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.840342 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6"] Nov 02 22:25:26.846089 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:25:26.846060 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c65b13b_f4dd_4a8a_9844_a0f0309be485.slice/crio-83645f9910503cfdf723d16b4db8395ed831fb72002892e1f2f55a69684e332c WatchSource:0}: Error finding container 83645f9910503cfdf723d16b4db8395ed831fb72002892e1f2f55a69684e332c: Status 404 returned error can't find the container with id 83645f9910503cfdf723d16b4db8395ed831fb72002892e1f2f55a69684e332c Nov 02 22:25:26.848818 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.848790 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:25:26.848896 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.848856 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:25:26.848932 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:26.848901 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:25:27.409245 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:27.409209 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:27.410795 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:27.410772 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 22:25:27.410857 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:27.410832 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" podUID="1c46dd84-f664-4317-86e5-035c682cd8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 22:25:27.795958 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:27.794979 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" event={"ID":"5c65b13b-f4dd-4a8a-9844-a0f0309be485","Type":"ContainerStarted","Data":"848f78054297f0d31c65fc30cdd75611291fe6264583b21b81ba90a9b5f7baba"} Nov 02 22:25:27.795958 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:27.795044 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" event={"ID":"5c65b13b-f4dd-4a8a-9844-a0f0309be485","Type":"ContainerStarted","Data":"83645f9910503cfdf723d16b4db8395ed831fb72002892e1f2f55a69684e332c"} Nov 02 22:25:27.912808 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:27.912743 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" podStartSLOduration=2.912728968 podStartE2EDuration="2.912728968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:26.87832045 +0000 UTC m=+1149.690650630" watchObservedRunningTime="2026-11-02 22:25:27.912728968 +0000 UTC m=+1150.725059154" Nov 02 22:25:27.915592 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:27.915540 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" podStartSLOduration=1.9155294459999999 podStartE2EDuration="1.915529446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:27.908776079 +0000 UTC m=+1150.721106291" watchObservedRunningTime="2026-11-02 22:25:27.915529446 +0000 UTC m=+1150.727859623" Nov 02 22:25:28.406903 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:28.406099 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:28.408025 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:28.407978 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 22:25:28.408200 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:28.408045 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" podUID="5c65b13b-f4dd-4a8a-9844-a0f0309be485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 22:25:28.408495 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:28.408470 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 22:25:28.408583 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:28.408508 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" podUID="1c46dd84-f664-4317-86e5-035c682cd8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 22:25:29.406287 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:29.406246 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 22:25:29.406688 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:29.406310 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" podUID="5c65b13b-f4dd-4a8a-9844-a0f0309be485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 22:25:29.409107 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:29.409084 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 22:25:29.409166 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:29.409130 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" podUID="1c46dd84-f664-4317-86e5-035c682cd8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 22:25:29.499914 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:29.496466 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:30.406063 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:30.405981 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 22:25:30.406063 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:30.406040 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" podUID="5c65b13b-f4dd-4a8a-9844-a0f0309be485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 22:25:30.409440 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:30.409410 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 22:25:30.409790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:30.409490 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" podUID="1c46dd84-f664-4317-86e5-035c682cd8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 22:25:31.406899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:31.406686 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 22:25:31.406899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:31.406744 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" podUID="5c65b13b-f4dd-4a8a-9844-a0f0309be485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 22:25:31.409386 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:31.409356 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 22:25:31.409523 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:31.409419 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" podUID="1c46dd84-f664-4317-86e5-035c682cd8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 22:25:32.405622 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:32.405583 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 22:25:32.405808 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:32.405650 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" podUID="5c65b13b-f4dd-4a8a-9844-a0f0309be485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 22:25:32.408759 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:32.408729 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 22:25:32.408905 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:32.408777 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" podUID="1c46dd84-f664-4317-86e5-035c682cd8da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 22:25:33.405694 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:33.405647 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 22:25:33.406109 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:33.405723 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" podUID="5c65b13b-f4dd-4a8a-9844-a0f0309be485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 22:25:33.709985 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:33.709853 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:33.710412 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:33.710381 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:33.711368 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:33.711341 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:25:33.734773 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:33.734730 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:25:34.367388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:34.367348 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:34.902123 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:34.902084 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:35.455016 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:35.454977 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:25:35.455353 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:35.455321 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" podUID="5c65b13b-f4dd-4a8a-9844-a0f0309be485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:25:36.315565 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:36.315525 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:36.317798 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:36.317771 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:36.319247 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:36.319224 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:25:40.168626 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:40.168587 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:43.084810 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:43.084772 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:43.770236 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:25:43.770199 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:25:45.093903 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:45.093843 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:49.908279 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:49.908241 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:54.700064 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:54.699936 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:25:54.997310 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:54.997224 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck"] Nov 02 22:25:55.002727 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.002703 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.091096 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.091063 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck"] Nov 02 22:25:55.157211 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.157175 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.157370 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.157290 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.157370 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.157351 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.157503 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.157391 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqjtx\" (UniqueName: \"kubernetes.io/projected/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-kube-api-access-xqjtx\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.157503 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.157451 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.157584 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.157541 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.157584 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.157570 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.157698 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.157682 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.157733 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.157707 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.258848 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.258744 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.258848 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.258796 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.259129 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.258946 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.259129 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.259032 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.259248 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.259123 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.259306 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.259229 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-workload-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.259364 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.259331 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqjtx\" (UniqueName: \"kubernetes.io/projected/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-kube-api-access-xqjtx\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.259426 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.259363 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.259426 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.259420 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.259537 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.259484 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.259636 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.259618 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-data\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.260277 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.259964 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istiod-ca-cert\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.260277 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.259999 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-workload-certs\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.260277 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.260141 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-credential-socket\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.262480 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.262454 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-envoy\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.263462 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.263441 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-podinfo\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.317060 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.317022 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-token\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.317215 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.317104 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqjtx\" (UniqueName: \"kubernetes.io/projected/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-kube-api-access-xqjtx\") pod \"e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.567490 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.567406 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2"] Nov 02 22:25:55.573764 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.573737 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.617730 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.617683 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:55.632182 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.632102 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2"] Nov 02 22:25:55.665356 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.665303 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.665552 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.665390 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e364f2d9-c537-4289-b605-547836336c31-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.665552 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.665424 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.665552 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.665514 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.665799 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.665630 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.665799 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.665679 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.665799 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.665723 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5tdf\" (UniqueName: \"kubernetes.io/projected/e364f2d9-c537-4289-b605-547836336c31-kube-api-access-h5tdf\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.666019 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.665799 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e364f2d9-c537-4289-b605-547836336c31-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.666019 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.665823 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e364f2d9-c537-4289-b605-547836336c31-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.767546 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.767513 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e364f2d9-c537-4289-b605-547836336c31-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.767996 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.767552 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.767996 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.767593 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.767996 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.767673 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.767996 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.767709 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.767996 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.767884 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5tdf\" (UniqueName: \"kubernetes.io/projected/e364f2d9-c537-4289-b605-547836336c31-kube-api-access-h5tdf\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.767996 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.767958 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e364f2d9-c537-4289-b605-547836336c31-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.767996 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.767977 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e364f2d9-c537-4289-b605-547836336c31-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.768362 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.768107 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.768362 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.768159 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.768362 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.768188 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.768362 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.768219 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.768541 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.768524 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.768856 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.768834 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e364f2d9-c537-4289-b605-547836336c31-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.770089 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.770072 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.770353 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.770334 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e364f2d9-c537-4289-b605-547836336c31-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.825415 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.825337 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e364f2d9-c537-4289-b605-547836336c31-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.825566 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.825513 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5tdf\" (UniqueName: \"kubernetes.io/projected/e364f2d9-c537-4289-b605-547836336c31-kube-api-access-h5tdf\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:55.890086 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:55.890053 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:56.058617 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.058558 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck"] Nov 02 22:25:56.063600 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:25:56.063569 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9d9f83_67f8_44dd_89a6_1c622fd4708a.slice/crio-68b52a14f9633c7b3704e7e5eba20a524b858a63b5f7929b4b24466169a8a5df WatchSource:0}: Error finding container 68b52a14f9633c7b3704e7e5eba20a524b858a63b5f7929b4b24466169a8a5df: Status 404 returned error can't find the container with id 68b52a14f9633c7b3704e7e5eba20a524b858a63b5f7929b4b24466169a8a5df Nov 02 22:25:56.066183 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.066150 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:25:56.066299 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.066231 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:25:56.066299 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.066276 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:25:56.312077 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.312046 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2"] Nov 02 22:25:56.317269 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:25:56.317239 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode364f2d9_c537_4289_b605_547836336c31.slice/crio-c851d59e12b590aa24cc2d84a93d5cf0a4bccbc1e47629c518154b476d685811 WatchSource:0}: Error finding container c851d59e12b590aa24cc2d84a93d5cf0a4bccbc1e47629c518154b476d685811: Status 404 returned error can't find the container with id c851d59e12b590aa24cc2d84a93d5cf0a4bccbc1e47629c518154b476d685811 Nov 02 22:25:56.320663 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.320624 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:25:56.320775 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.320714 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:25:56.320775 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.320756 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:25:56.596474 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.595859 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5"] Nov 02 22:25:56.600676 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.600655 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.678898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.678259 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5"] Nov 02 22:25:56.686841 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.686790 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.687106 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.686927 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.687106 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.686999 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.687106 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.687047 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.687106 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.687092 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3782bc26-cb56-41c1-b1d2-608d185145fc-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.687343 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.687177 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.687343 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.687226 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.687343 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.687305 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jtvd\" (UniqueName: \"kubernetes.io/projected/3782bc26-cb56-41c1-b1d2-608d185145fc-kube-api-access-5jtvd\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.689022 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.687364 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.788285 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.788249 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.788814 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.788409 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jtvd\" (UniqueName: \"kubernetes.io/projected/3782bc26-cb56-41c1-b1d2-608d185145fc-kube-api-access-5jtvd\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.788814 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.788472 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.788814 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.788539 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.788814 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.788627 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.788814 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.788686 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.788814 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.788733 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.788814 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.788735 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.788814 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.788775 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3782bc26-cb56-41c1-b1d2-608d185145fc-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.789347 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.788860 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.789347 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.788857 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.789347 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.789100 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.789509 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.789349 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.789791 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.789760 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3782bc26-cb56-41c1-b1d2-608d185145fc-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.791403 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.791374 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.791921 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.791887 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.846215 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.846182 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.846367 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.846246 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jtvd\" (UniqueName: \"kubernetes.io/projected/3782bc26-cb56-41c1-b1d2-608d185145fc-kube-api-access-5jtvd\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:56.923365 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:56.923329 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:57.110857 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:57.110821 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" event={"ID":"2e9d9f83-67f8-44dd-89a6-1c622fd4708a","Type":"ContainerStarted","Data":"3f3581db6a9bcbb3fa4a17d62a72b29243cd76f271540e47b13655f5f3e29936"} Nov 02 22:25:57.111151 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:57.111129 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" event={"ID":"2e9d9f83-67f8-44dd-89a6-1c622fd4708a","Type":"ContainerStarted","Data":"68b52a14f9633c7b3704e7e5eba20a524b858a63b5f7929b4b24466169a8a5df"} Nov 02 22:25:57.113345 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:57.113317 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" event={"ID":"e364f2d9-c537-4289-b605-547836336c31","Type":"ContainerStarted","Data":"21f2b004f6461e7a425715bbb1dfadac18c34ebb6183485a9684596f87be956c"} Nov 02 22:25:57.113475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:57.113366 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" event={"ID":"e364f2d9-c537-4289-b605-547836336c31","Type":"ContainerStarted","Data":"c851d59e12b590aa24cc2d84a93d5cf0a4bccbc1e47629c518154b476d685811"} Nov 02 22:25:57.230601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:57.230497 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" podStartSLOduration=3.230482619 podStartE2EDuration="3.230482619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:57.227990183 +0000 UTC m=+1180.040320360" watchObservedRunningTime="2026-11-02 22:25:57.230482619 +0000 UTC m=+1180.042812798" Nov 02 22:25:57.408317 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:57.408239 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" podStartSLOduration=2.408213789 podStartE2EDuration="2.408213789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:57.392640641 +0000 UTC m=+1180.204970822" watchObservedRunningTime="2026-11-02 22:25:57.408213789 +0000 UTC m=+1180.220543971" Nov 02 22:25:57.416689 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:57.416330 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5"] Nov 02 22:25:57.419227 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:25:57.419200 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3782bc26_cb56_41c1_b1d2_608d185145fc.slice/crio-80d1be7855a0f1ec4f7ffb6473a7c0ae8012b31fe355d6e30466fe65305480d6 WatchSource:0}: Error finding container 80d1be7855a0f1ec4f7ffb6473a7c0ae8012b31fe355d6e30466fe65305480d6: Status 404 returned error can't find the container with id 80d1be7855a0f1ec4f7ffb6473a7c0ae8012b31fe355d6e30466fe65305480d6 Nov 02 22:25:57.421928 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:57.421889 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:25:57.422031 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:57.421962 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:25:57.422031 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:57.421990 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:25:57.619237 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:57.619200 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:25:57.620894 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:57.620855 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 22:25:57.620959 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:57.620922 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" podUID="2e9d9f83-67f8-44dd-89a6-1c622fd4708a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 22:25:57.894899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:57.893944 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:25:57.901896 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:57.899413 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 22:25:57.901896 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:57.899492 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" podUID="e364f2d9-c537-4289-b605-547836336c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 22:25:58.123402 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:58.123359 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" event={"ID":"3782bc26-cb56-41c1-b1d2-608d185145fc","Type":"ContainerStarted","Data":"62a4239770c3f7918be6f2894c12468c36ea33ea38c646c8e31fe68a6fab1eb0"} Nov 02 22:25:58.123560 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:58.123422 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" event={"ID":"3782bc26-cb56-41c1-b1d2-608d185145fc","Type":"ContainerStarted","Data":"80d1be7855a0f1ec4f7ffb6473a7c0ae8012b31fe355d6e30466fe65305480d6"} Nov 02 22:25:58.279958 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:58.271228 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" podStartSLOduration=2.271207768 podStartE2EDuration="2.271207768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:58.268262411 +0000 UTC m=+1181.080592591" watchObservedRunningTime="2026-11-02 22:25:58.271207768 +0000 UTC m=+1181.083537948" Nov 02 22:25:58.637898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:58.636321 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 22:25:58.637898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:58.636387 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" podUID="2e9d9f83-67f8-44dd-89a6-1c622fd4708a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 22:25:58.904954 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:58.904585 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 22:25:58.904954 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:58.904649 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" podUID="e364f2d9-c537-4289-b605-547836336c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 22:25:58.929907 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:58.929784 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:25:58.932173 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:58.931289 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 22:25:58.932173 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:58.931345 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" podUID="3782bc26-cb56-41c1-b1d2-608d185145fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 22:25:59.627899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:59.627556 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 22:25:59.627899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:59.627620 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" podUID="2e9d9f83-67f8-44dd-89a6-1c622fd4708a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 22:25:59.898958 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:59.898119 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 22:25:59.898958 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:59.898181 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" podUID="e364f2d9-c537-4289-b605-547836336c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 22:25:59.924908 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:59.924813 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 22:25:59.924908 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:25:59.924891 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" podUID="3782bc26-cb56-41c1-b1d2-608d185145fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 22:26:00.252937 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:00.252817 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:00.624933 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:00.619269 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 22:26:00.624933 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:00.619351 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" podUID="2e9d9f83-67f8-44dd-89a6-1c622fd4708a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 22:26:00.894956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:00.894329 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 22:26:00.894956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:00.894415 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" podUID="e364f2d9-c537-4289-b605-547836336c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 22:26:00.924895 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:00.924390 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 22:26:00.924895 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:00.924482 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" podUID="3782bc26-cb56-41c1-b1d2-608d185145fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 22:26:01.622906 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:01.622336 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 22:26:01.622906 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:01.622405 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" podUID="2e9d9f83-67f8-44dd-89a6-1c622fd4708a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 22:26:01.892959 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:01.892290 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 22:26:01.892959 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:01.892373 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" podUID="e364f2d9-c537-4289-b605-547836336c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 22:26:01.930899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:01.929302 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 22:26:01.930899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:01.929365 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" podUID="3782bc26-cb56-41c1-b1d2-608d185145fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 22:26:02.619132 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:02.619092 5871 patch_prober.go:29] interesting pod/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 22:26:02.619330 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:02.619161 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" podUID="2e9d9f83-67f8-44dd-89a6-1c622fd4708a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 22:26:02.891212 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:02.891121 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 22:26:02.891212 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:02.891177 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" podUID="e364f2d9-c537-4289-b605-547836336c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 22:26:02.924506 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:02.924474 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 22:26:02.924676 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:02.924527 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" podUID="3782bc26-cb56-41c1-b1d2-608d185145fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 22:26:03.869901 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:03.869164 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:26:03.870421 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:03.870328 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:26:03.871538 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:03.871486 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:26:03.924784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:03.924746 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 22:26:03.924999 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:03.924826 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" podUID="3782bc26-cb56-41c1-b1d2-608d185145fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 22:26:04.068333 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:04.068291 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:04.169609 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:04.169529 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:26:04.170606 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:04.170584 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:26:04.175406 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:04.175383 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:26:05.224948 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:05.224914 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:26:05.225841 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:05.225821 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:26:05.226594 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:05.226568 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:26:05.389591 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:05.389556 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:07.403487 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:07.403453 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:07.584383 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:07.584329 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:10.310354 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.310290 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:10.708863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.708826 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29"] Nov 02 22:26:10.713477 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.713455 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.806200 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.806159 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2278f047-8cff-4d4f-b799-5b50740987c8-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.806544 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.806508 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.806668 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.806566 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.806902 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.806744 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rrqs\" (UniqueName: \"kubernetes.io/projected/2278f047-8cff-4d4f-b799-5b50740987c8-kube-api-access-5rrqs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.806902 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.806828 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.806902 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.806864 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2278f047-8cff-4d4f-b799-5b50740987c8-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.807207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.807032 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.807207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.807169 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2278f047-8cff-4d4f-b799-5b50740987c8-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.807320 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.807284 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.819183 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.816260 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29"] Nov 02 22:26:10.911004 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.910952 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.911209 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.911030 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.911209 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.911132 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rrqs\" (UniqueName: \"kubernetes.io/projected/2278f047-8cff-4d4f-b799-5b50740987c8-kube-api-access-5rrqs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.911209 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.911198 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.912239 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.911239 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2278f047-8cff-4d4f-b799-5b50740987c8-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.912239 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.911304 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.912239 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.911379 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2278f047-8cff-4d4f-b799-5b50740987c8-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.912239 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.911441 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.912239 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.911512 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2278f047-8cff-4d4f-b799-5b50740987c8-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.912501 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.912396 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-workload-certs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.912703 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.912681 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-workload-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.932897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.913549 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-istio-data\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.932897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.914255 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2278f047-8cff-4d4f-b799-5b50740987c8-istiod-ca-cert\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.932897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.924832 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-credential-socket\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.941273 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.941235 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-istio-envoy\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.945652 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.945461 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2278f047-8cff-4d4f-b799-5b50740987c8-istio-podinfo\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.971380 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.971299 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2278f047-8cff-4d4f-b799-5b50740987c8-istio-token\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:10.971513 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:10.971473 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rrqs\" (UniqueName: \"kubernetes.io/projected/2278f047-8cff-4d4f-b799-5b50740987c8-kube-api-access-5rrqs\") pod \"e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:11.027584 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:11.027542 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:11.548100 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:11.548059 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29"] Nov 02 22:26:11.624839 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:11.624788 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:26:11.625131 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:11.625106 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:26:11.625296 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:11.625275 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:26:12.535899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:12.535413 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" event={"ID":"2278f047-8cff-4d4f-b799-5b50740987c8","Type":"ContainerStarted","Data":"7d439cac0be14e875b5917d648619cdb786edc32e7344856911c03c9e28532eb"} Nov 02 22:26:12.535899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:12.535490 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" event={"ID":"2278f047-8cff-4d4f-b799-5b50740987c8","Type":"ContainerStarted","Data":"97f56dc38a202ccec3d48c5c6c27a91ac22d36b8c7527d563d42281b1514621a"} Nov 02 22:26:12.749899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:12.738974 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:12.818420 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:12.817834 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 22:26:12.843905 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:12.837044 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Nov 02 22:26:12.909123 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:12.909069 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:13.032994 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:13.032957 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:26:13.033255 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:13.033224 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" podUID="2278f047-8cff-4d4f-b799-5b50740987c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 22:26:13.033651 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:13.033630 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:13.053587 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:13.053541 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:13.132911 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:13.132848 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:13.211064 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:13.211022 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:13.251056 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:13.251007 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:13.411709 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:13.410225 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 22:26:13.411709 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:13.410655 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded" Nov 02 22:26:13.531001 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:13.530957 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:13.824737 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:26:13.824638 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:26:14.028554 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:14.028514 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:26:14.028735 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:14.028570 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" podUID="2278f047-8cff-4d4f-b799-5b50740987c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 22:26:15.028183 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:15.028148 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:26:15.028619 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:15.028205 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" podUID="2278f047-8cff-4d4f-b799-5b50740987c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 22:26:15.609592 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:15.609482 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:16.028056 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:16.028020 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:26:16.028244 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:16.028076 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" podUID="2278f047-8cff-4d4f-b799-5b50740987c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 22:26:17.028313 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:17.028279 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:26:17.028740 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:17.028336 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" podUID="2278f047-8cff-4d4f-b799-5b50740987c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 22:26:18.028318 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:18.028281 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:26:18.028726 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:18.028340 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" podUID="2278f047-8cff-4d4f-b799-5b50740987c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 22:26:19.028068 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:19.028027 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:26:19.028252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:19.028086 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" podUID="2278f047-8cff-4d4f-b799-5b50740987c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 22:26:20.129749 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:20.129707 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:20.130239 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:20.130005 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:20.130809 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:20.130782 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:26:20.294159 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:20.294097 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" podStartSLOduration=10.294082323 podStartE2EDuration="10.294082323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:12.778180297 +0000 UTC m=+1195.590510478" watchObservedRunningTime="2026-11-02 22:26:20.294082323 +0000 UTC m=+1203.106412492" Nov 02 22:26:20.576853 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:20.576752 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:24.923000 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:24.922965 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7"] Nov 02 22:26:24.928645 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:24.928616 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:24.999950 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:24.999911 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.000139 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:24.999966 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.000139 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.000069 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn5bg\" (UniqueName: \"kubernetes.io/projected/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-kube-api-access-rn5bg\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.000139 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.000128 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.000273 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.000157 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.000273 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.000239 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.000389 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.000362 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.000554 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.000538 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.000599 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.000576 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.101760 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.101721 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.101982 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.101778 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.101982 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.101804 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn5bg\" (UniqueName: \"kubernetes.io/projected/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-kube-api-access-rn5bg\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.101982 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.101845 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.101982 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.101907 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.101982 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.101941 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.102239 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.101992 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.102239 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.102128 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.102239 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.102162 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.102378 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.102343 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-data\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.102497 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.102471 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-workload-certs\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.102574 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.102504 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-workload-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.102701 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.102677 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-credential-socket\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.102787 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.102769 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istiod-ca-cert\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.104346 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.104320 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-envoy\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.104842 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.104814 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-podinfo\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.134931 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.134839 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7"] Nov 02 22:26:25.160176 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.160135 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-token\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.160479 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.160442 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn5bg\" (UniqueName: \"kubernetes.io/projected/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-kube-api-access-rn5bg\") pod \"e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.243140 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.243045 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:25.450164 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.450121 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:25.681770 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.681715 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7"] Nov 02 22:26:25.686928 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:26:25.686898 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01dbb32b_088f_4cb3_8c0e_c845fcefc0ce.slice/crio-c5da3131e5109725cbc713ba65fb414c628df408ea54b51f9fb5953f7531e48d WatchSource:0}: Error finding container c5da3131e5109725cbc713ba65fb414c628df408ea54b51f9fb5953f7531e48d: Status 404 returned error can't find the container with id c5da3131e5109725cbc713ba65fb414c628df408ea54b51f9fb5953f7531e48d Nov 02 22:26:25.689507 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.689471 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:26:25.689616 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.689554 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:26:25.689616 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:25.689599 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:26:26.182203 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.182141 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6"] Nov 02 22:26:26.187429 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.187398 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.249707 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.249654 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6"] Nov 02 22:26:26.322488 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.322445 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.322729 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.322497 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.322729 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.322652 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.322729 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.322719 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d38074-6059-4106-96a0-46fff0b1663c-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.322972 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.322916 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d38074-6059-4106-96a0-46fff0b1663c-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.323089 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.323066 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85rlx\" (UniqueName: \"kubernetes.io/projected/f2d38074-6059-4106-96a0-46fff0b1663c-kube-api-access-85rlx\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.323176 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.323158 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.323238 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.323203 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.323298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.323252 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d38074-6059-4106-96a0-46fff0b1663c-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.424580 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.424533 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85rlx\" (UniqueName: \"kubernetes.io/projected/f2d38074-6059-4106-96a0-46fff0b1663c-kube-api-access-85rlx\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.424766 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.424689 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.424766 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.424733 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.424957 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.424783 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d38074-6059-4106-96a0-46fff0b1663c-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.425022 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.424953 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.425022 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.424981 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.425133 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.425088 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.425201 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.425141 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d38074-6059-4106-96a0-46fff0b1663c-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.425258 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.425219 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d38074-6059-4106-96a0-46fff0b1663c-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.425309 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.425250 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.425309 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.425273 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.425601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.425572 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.425712 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.425608 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.426011 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.425988 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d38074-6059-4106-96a0-46fff0b1663c-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.427432 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.427412 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.427664 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.427639 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d38074-6059-4106-96a0-46fff0b1663c-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.482956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.482702 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85rlx\" (UniqueName: \"kubernetes.io/projected/f2d38074-6059-4106-96a0-46fff0b1663c-kube-api-access-85rlx\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.483125 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.483011 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d38074-6059-4106-96a0-46fff0b1663c-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.511212 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.511176 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:26.687373 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.687314 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" event={"ID":"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce","Type":"ContainerStarted","Data":"c3a4067a89c0cdd9b73ffda93e12d0824925e309ac439043c0df2d12cb0522c6"} Nov 02 22:26:26.687547 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.687398 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" event={"ID":"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce","Type":"ContainerStarted","Data":"c5da3131e5109725cbc713ba65fb414c628df408ea54b51f9fb5953f7531e48d"} Nov 02 22:26:26.804380 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.804268 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" podStartSLOduration=2.804252254 podStartE2EDuration="2.804252254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:26.800928194 +0000 UTC m=+1209.613258372" watchObservedRunningTime="2026-11-02 22:26:26.804252254 +0000 UTC m=+1209.616582432" Nov 02 22:26:26.984222 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.984192 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6"] Nov 02 22:26:26.985824 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:26:26.985783 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d38074_6059_4106_96a0_46fff0b1663c.slice/crio-f95326547f8270c1846922b7da88058a90bf9398d5efdcbece7d7ffce0722e67 WatchSource:0}: Error finding container f95326547f8270c1846922b7da88058a90bf9398d5efdcbece7d7ffce0722e67: Status 404 returned error can't find the container with id f95326547f8270c1846922b7da88058a90bf9398d5efdcbece7d7ffce0722e67 Nov 02 22:26:26.989235 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.989198 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:26:26.989355 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.989291 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:26:26.989355 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:26.989332 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:26:27.243740 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:27.243710 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:27.245118 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:27.245090 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 22:26:27.245249 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:27.245144 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" podUID="01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 22:26:27.716306 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:27.716263 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" event={"ID":"f2d38074-6059-4106-96a0-46fff0b1663c","Type":"ContainerStarted","Data":"a400ae5a3d4b1c1102bf86484bf49d3f02dac1f293d9e47aff81976e48878abb"} Nov 02 22:26:27.716490 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:27.716329 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" event={"ID":"f2d38074-6059-4106-96a0-46fff0b1663c","Type":"ContainerStarted","Data":"f95326547f8270c1846922b7da88058a90bf9398d5efdcbece7d7ffce0722e67"} Nov 02 22:26:27.840697 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:27.840642 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" podStartSLOduration=1.8406257080000001 podStartE2EDuration="1.840625708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:27.836281567 +0000 UTC m=+1210.648611744" watchObservedRunningTime="2026-11-02 22:26:27.840625708 +0000 UTC m=+1210.652955886" Nov 02 22:26:28.244106 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:28.244068 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 22:26:28.244689 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:28.244660 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" podUID="01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 22:26:28.512574 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:28.511429 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:28.513006 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:28.512979 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 22:26:28.513142 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:28.513034 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" podUID="f2d38074-6059-4106-96a0-46fff0b1663c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 22:26:29.243953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:29.243915 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 22:26:29.244116 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:29.243977 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" podUID="01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 22:26:29.512355 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:29.512261 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 22:26:29.512355 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:29.512322 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" podUID="f2d38074-6059-4106-96a0-46fff0b1663c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 22:26:30.244082 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:30.244048 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 22:26:30.244276 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:30.244101 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" podUID="01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 22:26:30.511668 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:30.511573 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 22:26:30.511668 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:30.511634 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" podUID="f2d38074-6059-4106-96a0-46fff0b1663c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 22:26:30.732231 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:30.732199 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:31.243558 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:31.243523 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 22:26:31.243747 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:31.243576 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" podUID="01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 22:26:31.512080 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:31.511985 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 22:26:31.512080 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:31.512056 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" podUID="f2d38074-6059-4106-96a0-46fff0b1663c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 22:26:32.244465 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:32.244427 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 22:26:32.245006 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:32.244488 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" podUID="01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 22:26:32.511858 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:32.511763 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 22:26:32.511858 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:32.511832 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" podUID="f2d38074-6059-4106-96a0-46fff0b1663c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 22:26:33.267488 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:33.267451 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:33.268078 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:33.268045 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:33.269540 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:33.269519 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:26:33.512277 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:33.512236 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 22:26:33.512457 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:33.512295 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" podUID="f2d38074-6059-4106-96a0-46fff0b1663c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 22:26:33.909391 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:33.909353 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:34.533790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:34.533754 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:34.534303 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:34.534212 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:34.538301 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:34.538278 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:26:35.990966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:35.990913 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:40.877303 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:40.877261 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:42.779009 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:42.778964 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:26:43.444950 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:43.444913 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:43.887081 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:26:43.887031 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:26:45.379249 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:45.376123 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 22:26:45.626943 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:45.625979 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:45.730959 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:45.727976 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 02 22:26:45.744907 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:45.741972 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:45.814901 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:45.811466 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:45.844899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:45.842094 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:45.917900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:45.913571 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:46.002958 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:46.002201 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:46.043901 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:46.043038 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:46.136708 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:46.136657 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:46.139185 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:46.137633 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 22:26:46.190075 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:46.190006 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:46.401898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:46.390471 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:46.401898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:46.391249 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:26:46.516135 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:46.516091 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 22:26:46.657981 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:46.630065 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:46.657981 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:46.630850 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:26:46.757130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:46.756994 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:46.796964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:46.771965 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:26:46.816186 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:46.816123 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:46.832900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:46.823982 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:26:46.852064 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:46.851106 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:46.928566 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:46.858006 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:26:47.032444 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.031383 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 02 22:26:47.047898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.032769 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:26:47.090899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.065921 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:47.191961 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.161086 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:26:47.212707 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.192974 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:47.217892 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.216569 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 22:26:47.218397 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.218159 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 22:26:47.349507 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.349469 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:26:47.372099 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.372060 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:26:47.409898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.406036 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:26:47.473959 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.402018 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:47.626897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.625289 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 22:26:47.626897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.626105 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:26:47.663531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.663474 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:47.768467 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.768359 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 02 22:26:47.835899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.823983 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:47.835899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.826100 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:47.894906 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.889051 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:47.894906 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.889978 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:47.949276 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:47.949224 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 02 22:26:48.054116 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.054016 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:48.054416 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.054381 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" podUID="667069a7-93cf-42ef-92ec-e825e6f6fabd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.170414 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.170356 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.229080 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.229023 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.325174 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.249866 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.326127 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.251977 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 22:26:48.326669 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.288950 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Nov 02 22:26:48.326824 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.323068 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.359273 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.359232 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" podUID="57e8906d-5f46-4056-baa4-a1401866e0ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 02 22:26:48.440031 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.439987 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:48.440726 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.440690 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" podUID="c30a7ffb-0fbd-437c-af03-a89ed7449bd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.653525 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.653478 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.655001 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.654959 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.788715 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.788342 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 02 22:26:48.971744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.941289 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" podUID="6b7ba2e0-b642-4b20-896b-e79253436f13" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.34:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.971744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.942157 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 22:26:48.971744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.954992 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:48.971744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:48.955345 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 02 22:26:49.011220 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.008791 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:49.011220 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.008896 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" podUID="8c608b09-9551-4d42-91c6-f6cb1a38bab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:49.170964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.170918 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:49.345961 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.335161 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:49.367164 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.367120 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 22:26:49.372538 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.372491 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 22:26:49.394098 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.394053 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:49.394396 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.394365 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" podUID="9384cb9b-0f07-485e-9635-c7dc585a8b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:49.400017 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.399971 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:49.449782 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.449749 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:49.450435 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.450409 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" podUID="01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:49.681043 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.676988 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:49.811282 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.811224 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:49.889580 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.700437 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:49.965090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.962212 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:49.965090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.962563 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:49.965090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:49.962773 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Nov 02 22:26:50.188674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.188397 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:26:50.223688 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.223595 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 02 22:26:50.269027 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.268991 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:26:50.290233 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.290188 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" podUID="e364f2d9-c537-4289-b605-547836336c31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:50.292183 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.292139 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:50.302496 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.302466 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:26:50.306632 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.306610 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:26:50.357007 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.355978 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:26:50.357007 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.356587 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:26:50.368289 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.367601 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:50.389085 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.389051 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:26:50.396086 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.396058 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:26:50.408149 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.408119 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:26:50.432904 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.431281 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:50.462238 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.462206 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:26:50.481790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.481703 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:26:50.498034 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:50.498005 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:26:52.595910 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:52.595857 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:26:57.490028 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:26:57.489993 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:02.278598 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:02.278554 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:04.422909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:04.422833 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:07.192198 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.192150 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:07.460645 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.460557 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn"] Nov 02 22:27:07.464907 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.464848 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.514066 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.514027 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 22:27:07.514066 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.514057 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 22:27:07.514293 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.514069 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 22:27:07.514293 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.514069 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-p7cs8\"" Nov 02 22:27:07.526508 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.526465 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxk28\" (UniqueName: \"kubernetes.io/projected/8b20c5aa-5920-488b-a5ab-d22d40b7af57-kube-api-access-vxk28\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.526658 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.526533 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.526658 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.526644 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b20c5aa-5920-488b-a5ab-d22d40b7af57-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.526748 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.526692 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.526840 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.526822 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.526914 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.526899 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.526958 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.526941 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.564586 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.564548 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn"] Nov 02 22:27:07.628609 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.628564 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxk28\" (UniqueName: \"kubernetes.io/projected/8b20c5aa-5920-488b-a5ab-d22d40b7af57-kube-api-access-vxk28\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.628788 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.628633 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.628788 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.628695 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b20c5aa-5920-488b-a5ab-d22d40b7af57-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.628788 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.628746 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.628962 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.628819 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.628962 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.628913 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.629031 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.628970 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.629979 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.629948 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.631347 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.631320 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b20c5aa-5920-488b-a5ab-d22d40b7af57-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.631488 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.631468 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.631657 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.631637 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.631730 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.631699 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.686715 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.686679 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxk28\" (UniqueName: \"kubernetes.io/projected/8b20c5aa-5920-488b-a5ab-d22d40b7af57-kube-api-access-vxk28\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.686970 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.686716 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fnnvn\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:07.778984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:07.778894 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:08.280621 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:08.280579 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn"] Nov 02 22:27:08.285541 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:27:08.285512 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b20c5aa_5920_488b_a5ab_d22d40b7af57.slice/crio-992e822647b2441e2f67445150910eb0d72b5efccf72a0d386b11ec03ae513e7 WatchSource:0}: Error finding container 992e822647b2441e2f67445150910eb0d72b5efccf72a0d386b11ec03ae513e7: Status 404 returned error can't find the container with id 992e822647b2441e2f67445150910eb0d72b5efccf72a0d386b11ec03ae513e7 Nov 02 22:27:08.287681 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:08.287665 5871 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:27:09.250819 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:09.250779 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" event={"ID":"8b20c5aa-5920-488b-a5ab-d22d40b7af57","Type":"ContainerStarted","Data":"992e822647b2441e2f67445150910eb0d72b5efccf72a0d386b11ec03ae513e7"} Nov 02 22:27:11.530243 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.530205 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8"] Nov 02 22:27:11.538561 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.538534 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.604114 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.604074 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8"] Nov 02 22:27:11.604529 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.604504 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.604661 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.604633 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3210ef32-d6e6-47d3-b431-826aba457e12-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.604744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.604706 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3210ef32-d6e6-47d3-b431-826aba457e12-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.604828 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.604807 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.605017 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.604951 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.605017 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.604985 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.605201 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.605128 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3210ef32-d6e6-47d3-b431-826aba457e12-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.605201 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.605189 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.605387 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.605363 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlbzf\" (UniqueName: \"kubernetes.io/projected/3210ef32-d6e6-47d3-b431-826aba457e12-kube-api-access-wlbzf\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.706487 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.706447 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.706681 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.706565 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.706681 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.706596 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.706681 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.706638 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3210ef32-d6e6-47d3-b431-826aba457e12-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.706681 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.706673 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.706956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.706805 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlbzf\" (UniqueName: \"kubernetes.io/projected/3210ef32-d6e6-47d3-b431-826aba457e12-kube-api-access-wlbzf\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.706956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.706907 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-workload-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.706956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.706933 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.707112 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.707014 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3210ef32-d6e6-47d3-b431-826aba457e12-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.707112 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.707068 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3210ef32-d6e6-47d3-b431-826aba457e12-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.707547 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.707516 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3210ef32-d6e6-47d3-b431-826aba457e12-istiod-ca-cert\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.707762 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.707743 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-credential-socket\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.707839 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.707804 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-istio-data\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.708080 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.708028 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-workload-certs\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.710365 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.710336 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-istio-envoy\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.711157 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.711124 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3210ef32-d6e6-47d3-b431-826aba457e12-istio-podinfo\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.767965 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.767918 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3210ef32-d6e6-47d3-b431-826aba457e12-istio-token\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.769493 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.769465 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlbzf\" (UniqueName: \"kubernetes.io/projected/3210ef32-d6e6-47d3-b431-826aba457e12-kube-api-access-wlbzf\") pod \"e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.856597 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.856499 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:11.978323 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.978275 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r"] Nov 02 22:27:11.986508 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:11.986479 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.031304 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.031266 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r"] Nov 02 22:27:12.126772 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.126738 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:12.127598 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.127558 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.127784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.127755 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.127935 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.127912 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.128011 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.127980 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.128059 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.128017 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.128181 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.128164 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgbbx\" (UniqueName: \"kubernetes.io/projected/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-kube-api-access-mgbbx\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.128240 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.128192 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.128307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.128283 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.128364 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.128332 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.230078 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.230042 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.230256 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.230166 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.230256 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.230249 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.230387 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.230294 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.230387 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.230332 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.230498 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.230465 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbbx\" (UniqueName: \"kubernetes.io/projected/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-kube-api-access-mgbbx\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.230557 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.230498 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.230557 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.230540 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.230662 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.230573 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.230662 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.230647 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-workload-certs\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.231324 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.230861 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istiod-ca-cert\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.231324 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.231261 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-credential-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.231574 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.231533 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-data\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.231574 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.231551 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-workload-socket\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.237113 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.234713 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-podinfo\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.237113 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.237035 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-envoy\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.288945 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.288866 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-token\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.288945 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.288937 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbbx\" (UniqueName: \"kubernetes.io/projected/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-kube-api-access-mgbbx\") pod \"e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.315653 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.314931 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8"] Nov 02 22:27:12.328574 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.328550 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:12.329009 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.328967 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:27:12.329137 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.329094 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:27:12.329221 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.329139 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:27:12.614604 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.614568 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:27:12.615042 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.614645 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:27:12.784108 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.784075 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r"] Nov 02 22:27:12.789508 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:27:12.789478 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8e7b9c_7a3a_4c87_b8b9_ab242f263124.slice/crio-eb29c10835c90dca130dcab17c77d957a5528bf8e87e28c75b5b10eaab4790a4 WatchSource:0}: Error finding container eb29c10835c90dca130dcab17c77d957a5528bf8e87e28c75b5b10eaab4790a4: Status 404 returned error can't find the container with id eb29c10835c90dca130dcab17c77d957a5528bf8e87e28c75b5b10eaab4790a4 Nov 02 22:27:12.792416 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.792375 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:27:12.792516 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.792475 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:27:12.792554 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:12.792522 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:27:13.292788 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.292741 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" event={"ID":"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124","Type":"ContainerStarted","Data":"1c0f9bd93065eca2701608f7d51d24a849b905da08ad4809f623770a0ff73970"} Nov 02 22:27:13.293033 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.292846 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" event={"ID":"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124","Type":"ContainerStarted","Data":"eb29c10835c90dca130dcab17c77d957a5528bf8e87e28c75b5b10eaab4790a4"} Nov 02 22:27:13.294976 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.294945 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" event={"ID":"3210ef32-d6e6-47d3-b431-826aba457e12","Type":"ContainerStarted","Data":"2897bee7bd40bef91fee1c27ee963b886ccf2a9a8eb068b7a5176cd3f4c1d656"} Nov 02 22:27:13.295117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.294997 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" event={"ID":"3210ef32-d6e6-47d3-b431-826aba457e12","Type":"ContainerStarted","Data":"6192ea125c940e65b358b70a100ab588b1edd1981d27474e003b01194eb3f1dc"} Nov 02 22:27:13.297482 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.297452 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" event={"ID":"8b20c5aa-5920-488b-a5ab-d22d40b7af57","Type":"ContainerStarted","Data":"dc2826c458a2a0a6ca94f7a4e50090365fee6288ffccd542e2ca9b34c951c935"} Nov 02 22:27:13.297624 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.297605 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:13.299784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.299762 5871 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-fnnvn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:27:13.299896 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.299812 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" podUID="8b20c5aa-5920-488b-a5ab-d22d40b7af57" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:27:13.329793 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.329762 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:13.331290 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.331261 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 22:27:13.331416 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.331314 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" podUID="dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 22:27:13.408697 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.408641 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" podStartSLOduration=2.408626103 podStartE2EDuration="2.408626103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:13.406455461 +0000 UTC m=+1256.218785618" watchObservedRunningTime="2026-11-02 22:27:13.408626103 +0000 UTC m=+1256.220956357" Nov 02 22:27:13.538416 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.537455 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" podStartSLOduration=2.211951394 podStartE2EDuration="6.537434315s" totalImagesPullingTime="4.325482921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:13.53599009 +0000 UTC m=+1256.348320281" watchObservedRunningTime="2026-11-02 22:27:13.537434315 +0000 UTC m=+1256.349764493" Nov 02 22:27:13.655982 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.653231 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" podStartSLOduration=2.6532151280000003 podStartE2EDuration="2.653215128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:13.649622099 +0000 UTC m=+1256.461952277" watchObservedRunningTime="2026-11-02 22:27:13.653215128 +0000 UTC m=+1256.465545307" Nov 02 22:27:13.841372 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:27:13.841104 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:27:13.870898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.860964 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:13.870898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.861142 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 22:27:13.870898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:13.861194 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" podUID="3210ef32-d6e6-47d3-b431-826aba457e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 22:27:14.326688 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:14.326277 5871 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-fnnvn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:27:14.326688 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:14.326342 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" podUID="8b20c5aa-5920-488b-a5ab-d22d40b7af57" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:27:14.329988 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:14.329959 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 22:27:14.330184 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:14.330162 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" podUID="dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 22:27:14.862908 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:14.859404 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 22:27:14.862908 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:14.859497 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" podUID="3210ef32-d6e6-47d3-b431-826aba457e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 22:27:15.207838 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:15.207743 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk"] Nov 02 22:27:15.214571 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:15.214526 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" podUID="57f82336-d76e-4f2b-b8e7-cb248650a5aa" containerName="istio-proxy" containerID="cri-o://365837f871487a8016a9a0eb4b623ca7649512ba38651880712bb8502260e6c8" gracePeriod=30 Nov 02 22:27:15.238246 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:15.238212 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9"] Nov 02 22:27:15.238504 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:15.238438 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" podUID="76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" containerName="istio-proxy" containerID="cri-o://1fbf7ac78b8c975d7e4c2a88dc91db3d9aa12f27743bb7a8092a95b4a742a507" gracePeriod=30 Nov 02 22:27:15.333414 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:15.331136 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 22:27:15.333414 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:15.331198 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" podUID="dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 22:27:15.880898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:15.858327 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 22:27:15.880898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:15.858393 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" podUID="3210ef32-d6e6-47d3-b431-826aba457e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 22:27:16.329232 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:16.329139 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 22:27:16.329232 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:16.329193 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" podUID="dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 22:27:16.879900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:16.872249 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 22:27:16.879900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:16.879049 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" podUID="3210ef32-d6e6-47d3-b431-826aba457e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 22:27:17.330157 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:17.329946 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 22:27:17.330157 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:17.330028 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" podUID="dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 22:27:17.340681 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:17.340638 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:27:17.867912 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:17.861285 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 22:27:17.867912 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:17.861351 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" podUID="3210ef32-d6e6-47d3-b431-826aba457e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 22:27:18.329275 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:18.329191 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 22:27:18.329275 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:18.329254 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" podUID="dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 22:27:18.893731 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:18.893602 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 22:27:18.894284 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:18.893696 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" podUID="3210ef32-d6e6-47d3-b431-826aba457e12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 22:27:19.330161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:19.330066 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 22:27:19.330161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:19.330136 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" podUID="dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 22:27:19.416104 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:19.416055 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:20.238359 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.237855 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:20.238641 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.238365 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:20.242904 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.242868 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:27:20.480585 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.480459 5871 generic.go:356] "Generic (PLEG): container finished" podID="57f82336-d76e-4f2b-b8e7-cb248650a5aa" containerID="365837f871487a8016a9a0eb4b623ca7649512ba38651880712bb8502260e6c8" exitCode=0 Nov 02 22:27:20.480585 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.480534 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" event={"ID":"57f82336-d76e-4f2b-b8e7-cb248650a5aa","Type":"ContainerDied","Data":"365837f871487a8016a9a0eb4b623ca7649512ba38651880712bb8502260e6c8"} Nov 02 22:27:20.482977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.482952 5871 generic.go:356] "Generic (PLEG): container finished" podID="76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" containerID="1fbf7ac78b8c975d7e4c2a88dc91db3d9aa12f27743bb7a8092a95b4a742a507" exitCode=0 Nov 02 22:27:20.483991 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.483961 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" event={"ID":"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989","Type":"ContainerDied","Data":"1fbf7ac78b8c975d7e4c2a88dc91db3d9aa12f27743bb7a8092a95b4a742a507"} Nov 02 22:27:20.602811 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.602776 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:27:20.671901 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.670793 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:27:20.723290 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.723246 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-kube-api-access-w2jqm\" (UniqueName: \"kubernetes.io/projected/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-kube-api-access-w2jqm\") pod \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " Nov 02 22:27:20.723472 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.723318 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istiod-ca-cert\") pod \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " Nov 02 22:27:20.723472 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.723388 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-data\") pod \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " Nov 02 22:27:20.723472 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.723434 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-workload-certs\") pod \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " Nov 02 22:27:20.725424 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.723473 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f82336-d76e-4f2b-b8e7-cb248650a5aa-kube-api-access-wjxm4\" (UniqueName: \"kubernetes.io/projected/57f82336-d76e-4f2b-b8e7-cb248650a5aa-kube-api-access-wjxm4\") pod \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " Nov 02 22:27:20.725569 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.723767 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istiod-ca-cert" pod "57f82336-d76e-4f2b-b8e7-cb248650a5aa" (UID: "57f82336-d76e-4f2b-b8e7-cb248650a5aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:20.725569 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.725504 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-token\" (UniqueName: \"kubernetes.io/projected/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-token\") pod \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " Nov 02 22:27:20.725569 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.724038 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-data" pod "57f82336-d76e-4f2b-b8e7-cb248650a5aa" (UID: "57f82336-d76e-4f2b-b8e7-cb248650a5aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:20.725569 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.724318 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-workload-certs" pod "57f82336-d76e-4f2b-b8e7-cb248650a5aa" (UID: "57f82336-d76e-4f2b-b8e7-cb248650a5aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:20.725569 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.725554 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-envoy\") pod \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " Nov 02 22:27:20.725977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.725672 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-workload-socket\") pod \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " Nov 02 22:27:20.725977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.725733 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-data\") pod \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " Nov 02 22:27:20.725977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.725764 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istiod-ca-cert\") pod \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " Nov 02 22:27:20.725977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.725830 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-workload-socket\") pod \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " Nov 02 22:27:20.725977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.725885 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-workload-certs\") pod \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " Nov 02 22:27:20.725977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.725917 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-podinfo\") pod \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " Nov 02 22:27:20.725977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.725969 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-podinfo\") pod \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " Nov 02 22:27:20.726346 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.726092 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-workload-socket" pod "57f82336-d76e-4f2b-b8e7-cb248650a5aa" (UID: "57f82336-d76e-4f2b-b8e7-cb248650a5aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:20.726346 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.726108 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-envoy\") pod \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " Nov 02 22:27:20.726346 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.726140 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-credential-socket\") pod \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " Nov 02 22:27:20.726524 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.726364 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-workload-socket" pod "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" (UID: "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:20.726524 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.726363 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f82336-d76e-4f2b-b8e7-cb248650a5aa-kube-api-access-wjxm4" pod "57f82336-d76e-4f2b-b8e7-cb248650a5aa" (UID: "57f82336-d76e-4f2b-b8e7-cb248650a5aa"). InnerVolumeSpecName "kube-api-access-wjxm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:20.726634 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.726601 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-workload-certs" pod "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" (UID: "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:20.726946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.726752 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istiod-ca-cert" pod "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" (UID: "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:20.727159 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.727132 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-token\" (UniqueName: \"kubernetes.io/projected/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-token\") pod \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\" (UID: \"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989\") " Nov 02 22:27:20.727317 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.727228 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-credential-socket\") pod \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\" (UID: \"57f82336-d76e-4f2b-b8e7-cb248650a5aa\") " Nov 02 22:27:20.727772 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.727738 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-credential-socket" pod "57f82336-d76e-4f2b-b8e7-cb248650a5aa" (UID: "57f82336-d76e-4f2b-b8e7-cb248650a5aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:20.728033 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.728008 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-token" pod "57f82336-d76e-4f2b-b8e7-cb248650a5aa" (UID: "57f82336-d76e-4f2b-b8e7-cb248650a5aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:20.728166 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.728110 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-data" pod "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" (UID: "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:20.728307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.728282 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-credential-socket" pod "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" (UID: "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:20.728404 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.728381 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-kube-api-access-w2jqm" pod "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" (UID: "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989"). InnerVolumeSpecName "kube-api-access-w2jqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:20.730966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.728755 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-podinfo" pod "57f82336-d76e-4f2b-b8e7-cb248650a5aa" (UID: "57f82336-d76e-4f2b-b8e7-cb248650a5aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:27:20.730966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.730155 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.730966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.730182 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.730966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.730197 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2jqm\" (UniqueName: \"kubernetes.io/projected/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-kube-api-access-w2jqm\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.730966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.730214 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.730966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.730230 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.730966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.730245 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.730966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.730260 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjxm4\" (UniqueName: \"kubernetes.io/projected/57f82336-d76e-4f2b-b8e7-cb248650a5aa-kube-api-access-wjxm4\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.730966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.730274 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.730966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.730290 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.730966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.730302 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.730966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.730316 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.730966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.730329 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.730966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.730343 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.730966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.730356 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.730966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.730481 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-envoy" pod "57f82336-d76e-4f2b-b8e7-cb248650a5aa" (UID: "57f82336-d76e-4f2b-b8e7-cb248650a5aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:20.731839 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.731244 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-token" pod "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" (UID: "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:20.731839 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.731736 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-podinfo" pod "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" (UID: "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:27:20.733207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.733183 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-envoy" pod "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" (UID: "76eb16d5-a9c1-4fe6-a3ef-0f195f29e989"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:20.751888 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.751840 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:20.752385 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.752297 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:20.753580 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.753510 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:27:20.831680 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.831639 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f82336-d76e-4f2b-b8e7-cb248650a5aa-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.831680 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.831671 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.831680 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.831685 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.831930 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:20.831697 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:21.492700 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:21.492672 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" Nov 02 22:27:21.493166 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:21.492666 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk" event={"ID":"57f82336-d76e-4f2b-b8e7-cb248650a5aa","Type":"ContainerDied","Data":"f300e89f1dc4bd8a8636be984824d9882091c2ce99cedc2dd088c7755016b4ae"} Nov 02 22:27:21.493166 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:21.492820 5871 scope.go:122] "RemoveContainer" containerID="365837f871487a8016a9a0eb4b623ca7649512ba38651880712bb8502260e6c8" Nov 02 22:27:21.494227 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:21.494158 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" Nov 02 22:27:21.494283 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:21.494157 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9" event={"ID":"76eb16d5-a9c1-4fe6-a3ef-0f195f29e989","Type":"ContainerDied","Data":"9cdf7edd970d22dd41d1fd1c7bfdfff592151d623f9625971621b8417d322e87"} Nov 02 22:27:21.504425 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:21.504405 5871 scope.go:122] "RemoveContainer" containerID="1fbf7ac78b8c975d7e4c2a88dc91db3d9aa12f27743bb7a8092a95b4a742a507" Nov 02 22:27:21.671932 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:21.665982 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk"] Nov 02 22:27:21.671932 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:21.671728 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f7qmk"] Nov 02 22:27:21.789439 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:21.789390 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9"] Nov 02 22:27:21.795833 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:21.795800 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4j9"] Nov 02 22:27:23.332743 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:23.332708 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f82336-d76e-4f2b-b8e7-cb248650a5aa" path="/var/lib/kubelet/pods/57f82336-d76e-4f2b-b8e7-cb248650a5aa/volumes" Nov 02 22:27:23.348124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:23.334091 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" path="/var/lib/kubelet/pods/76eb16d5-a9c1-4fe6-a3ef-0f195f29e989/volumes" Nov 02 22:27:24.268004 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:24.267959 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 02 22:27:24.413944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:24.413017 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:24.798906 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:24.790921 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 02 22:27:25.234671 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.227910 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:25.235661 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.235626 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:25.303196 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.272065 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 02 22:27:25.303196 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.272648 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:27:25.317672 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.307507 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:25.363965 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.339460 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:25.416046 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.415973 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:25.424686 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.424646 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:27:25.456470 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.456423 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 22:27:25.457601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.457568 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:25.481119 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.481071 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:25.490074 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.489962 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:25.692901 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.689745 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:25.743284 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.743176 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:25.864838 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.864792 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:25.866660 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.866632 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:27:25.961939 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:25.960862 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Nov 02 22:27:26.003114 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.003003 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:26.049235 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.049190 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:26.235906 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.229142 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:26.242912 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.241989 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:27:26.298086 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.297985 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:26.346013 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.345967 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 22:27:26.365198 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.346274 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 02 22:27:26.401899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.392121 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:27:26.445991 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.420662 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 22:27:26.450774 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.449816 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:27:26.459757 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.458207 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:26.472306 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.460071 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:27:26.472306 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.471709 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:26.498308 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.496976 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 22:27:26.519680 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.518670 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:27:26.761316 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.759993 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:26.768941 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.761666 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:27:26.852015 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.822460 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" podUID="6b7ba2e0-b642-4b20-896b-e79253436f13" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.34:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:26.866009 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.865962 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 02 22:27:26.906219 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.906073 5871 patch_prober.go:29] interesting pod/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:27:26.906219 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.906134 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" podUID="291b0b67-ee70-4d16-ac08-8064e7f0fef1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:26.967898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.967821 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Nov 02 22:27:26.968594 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:26.968570 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:27:27.016042 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:27.015940 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:27.089555 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:27.089520 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:27:27.328956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:27.291034 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 22:27:27.329275 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:27.329227 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Nov 02 22:27:27.508325 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:27.487965 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:27.514000 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:27.508917 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 22:27:27.514000 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:27.508947 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 22:27:27.517578 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:27.517543 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:27.535081 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:27.535012 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:27.589952 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:27.334193 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:27.634937 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:27.634841 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:27.775109 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:27.775062 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:28.019977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:27.914373 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded" Nov 02 22:27:28.056975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:28.020261 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:28.057241 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:28.053985 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:28.074559 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:28.074513 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:28.090261 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:28.090215 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:28.134693 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:28.134657 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:27:28.364961 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:28.364000 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:28.530846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:28.528009 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 22:27:28.530846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:28.528527 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 22:27:28.568914 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:28.550006 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:28.620956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:28.604018 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 02 22:27:28.620956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:28.604254 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:28.848441 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:28.848395 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" podUID="6b7ba2e0-b642-4b20-896b-e79253436f13" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.34:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:29.200901 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:29.194897 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 22:27:29.224467 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:29.224413 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:29.226083 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:29.226046 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:29.269898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:29.256555 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:29.399284 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:29.324057 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:29.400030 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:29.365257 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 22:27:29.534960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:29.531054 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:29.559299 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:29.531283 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:29.559786 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:29.550474 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:29.608034 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:29.607982 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:29.609391 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:29.608301 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:30.057900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.056408 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:27:30.108332 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.108301 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:27:30.175633 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.174726 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:27:30.240971 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.237902 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:30.240971 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.238276 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 22:27:30.240971 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.238605 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:27:30.314923 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.314776 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Nov 02 22:27:30.404633 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.404511 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:30.408990 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.408514 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:27:30.408990 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.408579 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:27:30.441072 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.419091 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:27:30.441072 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.422573 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:27:30.441072 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.429253 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:27:30.441769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.441743 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:27:30.507335 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.506857 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:27:30.510315 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.510290 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:27:30.552602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.552530 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:27:30.581990 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.581892 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:30.789897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:30.789830 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:31.210767 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:31.210732 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:33.489853 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:33.489810 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:33.674968 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:33.674931 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:33.854087 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:33.853978 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:34.265479 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.265438 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb"] Nov 02 22:27:34.266122 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.266107 5871 cpu_manager.go:414] "containerMap: removing container" podUID="57f82336-d76e-4f2b-b8e7-cb248650a5aa" containerName="istio-proxy" Nov 02 22:27:34.266193 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.266122 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f82336-d76e-4f2b-b8e7-cb248650a5aa" containerName="istio-proxy" Nov 02 22:27:34.266193 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.266141 5871 cpu_manager.go:414] "containerMap: removing container" podUID="76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" containerName="istio-proxy" Nov 02 22:27:34.266193 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.266146 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" containerName="istio-proxy" Nov 02 22:27:34.266289 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.266237 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="76eb16d5-a9c1-4fe6-a3ef-0f195f29e989" containerName="istio-proxy" Nov 02 22:27:34.266289 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.266255 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f82336-d76e-4f2b-b8e7-cb248650a5aa" containerName="istio-proxy" Nov 02 22:27:34.269131 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.269107 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.366538 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.366497 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb"] Nov 02 22:27:34.371330 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.371286 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istiod-ca-cert\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.371517 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.371344 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-token\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.371517 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.371473 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-podinfo\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.371618 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.371594 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-data\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.371660 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.371638 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-workload-socket\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.371815 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.371797 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v24s\" (UniqueName: \"kubernetes.io/projected/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-kube-api-access-8v24s\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.371857 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.371843 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-credential-socket\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.371981 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.371966 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-envoy\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.372022 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.372003 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-workload-certs\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.374550 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.374518 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7f1542e5-openshift-default-dockercfg-bhclq\"" Nov 02 22:27:34.436710 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.436667 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:34.480905 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.480848 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istiod-ca-cert\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.481213 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.481187 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-token\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.481384 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.481366 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-podinfo\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.481680 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.481642 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-data\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.481845 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.481827 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-workload-socket\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.482127 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.482109 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v24s\" (UniqueName: \"kubernetes.io/projected/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-kube-api-access-8v24s\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.482276 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.482261 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-credential-socket\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.482460 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.482445 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-envoy\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.482583 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.482567 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-workload-certs\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.483349 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.483326 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-workload-certs\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.484520 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.484489 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istiod-ca-cert\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.485199 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.485125 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-data\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.485418 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.485394 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-workload-socket\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.485670 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.485649 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-credential-socket\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.491935 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.487240 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-podinfo\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.564859 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.564779 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-envoy\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.569443 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.569410 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v24s\" (UniqueName: \"kubernetes.io/projected/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-kube-api-access-8v24s\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.580836 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.580796 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-token\") pod \"e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:34.584542 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:34.584506 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:35.046165 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:35.046128 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:35.048651 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:35.048624 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb"] Nov 02 22:27:35.050335 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:27:35.050309 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a975b1_786e_42dd_88fa_9bdd36a50ffe.slice/crio-b90b5f8780998c97cb301768b865a1fdb06b8e5a31d8d73fded513de2aa3f5d2 WatchSource:0}: Error finding container b90b5f8780998c97cb301768b865a1fdb06b8e5a31d8d73fded513de2aa3f5d2: Status 404 returned error can't find the container with id b90b5f8780998c97cb301768b865a1fdb06b8e5a31d8d73fded513de2aa3f5d2 Nov 02 22:27:35.053091 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:35.053059 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:27:35.053175 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:35.053132 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:27:35.053213 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:35.053174 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:27:35.455820 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:35.455722 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" event={"ID":"b2a975b1-786e-42dd-88fa-9bdd36a50ffe","Type":"ContainerStarted","Data":"37eeaa2c5b8da2e6bfbb7efe937577bcb38b71cc482938840bf53428c0646ce6"} Nov 02 22:27:35.455820 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:35.455810 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" event={"ID":"b2a975b1-786e-42dd-88fa-9bdd36a50ffe","Type":"ContainerStarted","Data":"b90b5f8780998c97cb301768b865a1fdb06b8e5a31d8d73fded513de2aa3f5d2"} Nov 02 22:27:35.586409 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:35.586334 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" podStartSLOduration=1.586316799 podStartE2EDuration="1.586316799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:35.585988338 +0000 UTC m=+1278.398318517" watchObservedRunningTime="2026-11-02 22:27:35.586316799 +0000 UTC m=+1278.398646977" Nov 02 22:27:35.589865 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:35.589833 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:36.592958 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:36.592927 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:37.376288 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:37.376244 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:37.472326 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:37.472286 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:37.473296 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:37.473274 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:27:37.968318 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:37.968280 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:38.165501 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:38.165465 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:38.358369 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:38.358281 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:39.722243 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:39.722198 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:39.895090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:39.895055 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:42.487091 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.487056 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9"] Nov 02 22:27:42.491486 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.491460 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.547321 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.547282 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9"] Nov 02 22:27:42.560941 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.560895 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa92502-a30d-4230-9338-80d73e06c66f-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.561130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.560965 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.561130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.561099 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.561235 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.561139 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa92502-a30d-4230-9338-80d73e06c66f-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.561270 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.561256 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.561319 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.561301 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72qg6\" (UniqueName: \"kubernetes.io/projected/3aa92502-a30d-4230-9338-80d73e06c66f-kube-api-access-72qg6\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.561478 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.561461 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa92502-a30d-4230-9338-80d73e06c66f-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.561619 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.561593 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.561709 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.561691 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.662659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.662613 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.662845 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.662684 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.662845 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.662755 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa92502-a30d-4230-9338-80d73e06c66f-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.662845 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.662783 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.662845 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.662819 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.662845 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.662841 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa92502-a30d-4230-9338-80d73e06c66f-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.663134 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.662942 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.663134 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.662986 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qg6\" (UniqueName: \"kubernetes.io/projected/3aa92502-a30d-4230-9338-80d73e06c66f-kube-api-access-72qg6\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.663134 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.663089 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa92502-a30d-4230-9338-80d73e06c66f-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.663410 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.663166 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-istio-data\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.663410 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.663316 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-workload-certs\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.663410 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.663366 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-workload-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.663586 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.663516 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-credential-socket\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.663824 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.663806 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa92502-a30d-4230-9338-80d73e06c66f-istiod-ca-cert\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.665411 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.665386 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-istio-envoy\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.665553 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.665534 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa92502-a30d-4230-9338-80d73e06c66f-istio-podinfo\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.724640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.724598 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa92502-a30d-4230-9338-80d73e06c66f-istio-token\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.724825 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.724789 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72qg6\" (UniqueName: \"kubernetes.io/projected/3aa92502-a30d-4230-9338-80d73e06c66f-kube-api-access-72qg6\") pod \"e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:42.810501 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:42.810409 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:43.254150 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:43.254121 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9"] Nov 02 22:27:43.255644 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:27:43.255606 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa92502_a30d_4230_9338_80d73e06c66f.slice/crio-aaf5cb8e0b1d5f2ff9c3f3f6750e794044225192d8e2889488d8282bf1a12a2b WatchSource:0}: Error finding container aaf5cb8e0b1d5f2ff9c3f3f6750e794044225192d8e2889488d8282bf1a12a2b: Status 404 returned error can't find the container with id aaf5cb8e0b1d5f2ff9c3f3f6750e794044225192d8e2889488d8282bf1a12a2b Nov 02 22:27:43.258950 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:43.258915 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:27:43.259039 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:43.258991 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:27:43.259039 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:43.259023 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:27:43.527125 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:43.527020 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" event={"ID":"3aa92502-a30d-4230-9338-80d73e06c66f","Type":"ContainerStarted","Data":"6c9d84c786093b37fc77810c778bf3587e7bf59710b414099c9faeaca1f6bc99"} Nov 02 22:27:43.527125 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:43.527080 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" event={"ID":"3aa92502-a30d-4230-9338-80d73e06c66f","Type":"ContainerStarted","Data":"aaf5cb8e0b1d5f2ff9c3f3f6750e794044225192d8e2889488d8282bf1a12a2b"} Nov 02 22:27:43.649020 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:43.648967 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" podStartSLOduration=1.64895144 podStartE2EDuration="1.64895144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:43.646807509 +0000 UTC m=+1286.459137688" watchObservedRunningTime="2026-11-02 22:27:43.64895144 +0000 UTC m=+1286.461281699" Nov 02 22:27:43.773622 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:27:43.773587 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:27:43.811137 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:43.810977 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:43.817412 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:43.817373 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:44.812637 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:44.812598 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:27:44.813231 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:44.812658 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" podUID="3aa92502-a30d-4230-9338-80d73e06c66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 22:27:45.433192 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:45.433157 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:45.624312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:45.624268 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:45.811691 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:45.811588 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:27:45.811910 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:45.811679 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" podUID="3aa92502-a30d-4230-9338-80d73e06c66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 22:27:45.831353 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:45.831318 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:46.811721 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:46.811679 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:27:46.812015 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:46.811746 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" podUID="3aa92502-a30d-4230-9338-80d73e06c66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 22:27:47.810777 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:47.810731 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:27:47.811344 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:47.810799 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" podUID="3aa92502-a30d-4230-9338-80d73e06c66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 22:27:48.811917 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:48.811831 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:27:48.812366 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:48.811934 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" podUID="3aa92502-a30d-4230-9338-80d73e06c66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 22:27:49.810916 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:49.810868 5871 patch_prober.go:29] interesting pod/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:27:49.811129 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:49.810952 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" podUID="3aa92502-a30d-4230-9338-80d73e06c66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 22:27:50.839492 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:50.839443 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:50.840501 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:50.840416 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:50.841205 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:50.841187 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:27:51.173089 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:51.173052 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:27:55.939241 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:27:55.939193 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:00.913568 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:00.913526 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:01.984427 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:01.984391 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:02.172901 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:02.172857 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:02.889892 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:02.889838 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:28:03.639082 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:03.638988 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:03.836964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:03.836917 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:04.031956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:04.030998 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:04.800109 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:04.758997 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:05.055858 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:05.055432 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:05.245906 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:05.245747 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:05.346954 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:05.326828 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:05.405898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:05.403956 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:05.431024 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:05.430966 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:05.433568 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:05.433530 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:05.434123 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:05.434095 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 22:28:05.493035 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:05.492978 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:05.518060 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:05.518011 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:05.552093 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:05.552026 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:05.741090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:05.741026 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:05.932018 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:05.931963 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:06.090955 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:06.089996 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:06.093002 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:06.092957 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:06.412937 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:06.409967 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:06.412937 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:06.410731 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:28:06.445894 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:06.439711 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:06.445894 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:06.440116 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:28:06.527009 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:06.526964 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:06.527933 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:06.527907 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:28:07.471900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:07.469030 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:07.702900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:07.652109 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:07.756954 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:07.664432 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 22:28:07.756954 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:07.664514 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 22:28:07.756954 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:07.664530 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:07.811391 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:07.683035 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:07.998900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:07.717029 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:08.102959 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.100107 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:08.217261 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.217222 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 02 22:28:08.426379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.217537 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 02 22:28:08.427440 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.217569 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Nov 02 22:28:08.427649 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.255125 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tmtfl" podUID="57e8906d-5f46-4056-baa4-a1401866e0ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:08.428058 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.335140 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:08.428451 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.394174 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:08.470247 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.470192 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:08.538910 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:28:08.499615 5871 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.143s" Nov 02 22:28:08.875957 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.780805 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:08.875957 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.780841 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 02 22:28:08.875957 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.862322 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:28:08.875957 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.862384 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:28:08.875957 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.780890 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 22:28:08.875957 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.813839 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:08.875957 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.826938 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:08.875957 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.863752 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:28:08.875957 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:08.863803 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:28:09.111076 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.111027 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:09.122899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.111707 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:28:09.343174 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.234906 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 02 22:28:09.448899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.236758 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:09.449305 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.362014 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z4rhr" podUID="6b7ba2e0-b642-4b20-896b-e79253436f13" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.34:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:09.456900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.430779 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 02 22:28:09.482348 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.432291 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 02 22:28:09.669585 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.432325 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 02 22:28:09.670418 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.495302 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 22:28:09.702412 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.699071 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:09.857896 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.853659 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 22:28:09.874374 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.874330 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:09.874974 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.874945 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 02 22:28:09.886894 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.875381 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 02 22:28:09.887808 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.880591 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:09.918136 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.918095 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:28:09.918412 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.918399 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:28:09.918544 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.918535 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:28:09.918653 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:09.918645 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:28:10.253333 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:10.253299 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:28:10.254260 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:10.254236 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:28:10.299419 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:10.299370 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:28:10.299717 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:10.299703 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:28:10.301154 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:10.301106 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:10.307698 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:10.302591 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:28:10.317867 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:10.317838 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:28:10.481288 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:10.481253 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:28:10.500907 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:10.500408 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" Nov 02 22:28:10.505665 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:10.503132 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:13.772853 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:28:13.772811 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:28:14.843733 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:14.843691 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:15.047783 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:15.047738 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:15.234194 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:15.234154 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:16.123884 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:16.123836 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:16.333653 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:16.333605 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:17.167278 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:17.167245 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:20.130319 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:20.130278 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:21.218687 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:21.218649 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr"] Nov 02 22:28:21.334714 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:21.334666 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:22.383113 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:22.383079 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:24.673976 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:24.673939 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:27.147728 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:27.147686 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:27.337734 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:27.337693 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:28.351079 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:28.351039 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:29.591837 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:29.591801 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:29.769173 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:29.769111 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:32.303054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:32.303014 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:32.953047 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:32.953006 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:33.132290 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:33.132248 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:35.094060 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:35.094006 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:36.037806 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.037766 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:36.241710 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.241622 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" containerID="cri-o://56c4dabe19fcafc8494fb96e1a0aded7347241adcb50973455eef756f7f83129" gracePeriod=45 Nov 02 22:28:36.533989 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.533960 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:28:36.656374 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.656340 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-tmp-dir\") pod \"c422c51d-7d39-4751-92c5-0de8ca3880a0\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " Nov 02 22:28:36.656570 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.656437 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c422c51d-7d39-4751-92c5-0de8ca3880a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/c422c51d-7d39-4751-92c5-0de8ca3880a0-tls-certs\") pod \"c422c51d-7d39-4751-92c5-0de8ca3880a0\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " Nov 02 22:28:36.656650 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.656634 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-model-cache\") pod \"c422c51d-7d39-4751-92c5-0de8ca3880a0\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " Nov 02 22:28:36.656726 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.656713 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-dshm\") pod \"c422c51d-7d39-4751-92c5-0de8ca3880a0\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " Nov 02 22:28:36.656929 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.656782 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-home\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-home\") pod \"c422c51d-7d39-4751-92c5-0de8ca3880a0\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " Nov 02 22:28:36.656929 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.656846 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c422c51d-7d39-4751-92c5-0de8ca3880a0-kube-api-access-55bfp\" (UniqueName: \"kubernetes.io/projected/c422c51d-7d39-4751-92c5-0de8ca3880a0-kube-api-access-55bfp\") pod \"c422c51d-7d39-4751-92c5-0de8ca3880a0\" (UID: \"c422c51d-7d39-4751-92c5-0de8ca3880a0\") " Nov 02 22:28:36.656929 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.656891 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-model-cache" pod "c422c51d-7d39-4751-92c5-0de8ca3880a0" (UID: "c422c51d-7d39-4751-92c5-0de8ca3880a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:28:36.657105 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.657081 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-home" pod "c422c51d-7d39-4751-92c5-0de8ca3880a0" (UID: "c422c51d-7d39-4751-92c5-0de8ca3880a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:28:36.657165 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.656670 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-tmp-dir" pod "c422c51d-7d39-4751-92c5-0de8ca3880a0" (UID: "c422c51d-7d39-4751-92c5-0de8ca3880a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:28:36.657723 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.657699 5871 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-model-cache\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:36.657961 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.657944 5871 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-home\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:36.658102 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.658088 5871 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-tmp-dir\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:36.667428 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.667374 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c422c51d-7d39-4751-92c5-0de8ca3880a0-tls-certs" pod "c422c51d-7d39-4751-92c5-0de8ca3880a0" (UID: "c422c51d-7d39-4751-92c5-0de8ca3880a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:28:36.687451 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.687413 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-dshm" pod "c422c51d-7d39-4751-92c5-0de8ca3880a0" (UID: "c422c51d-7d39-4751-92c5-0de8ca3880a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:28:36.689069 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.689029 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c422c51d-7d39-4751-92c5-0de8ca3880a0-kube-api-access-55bfp" pod "c422c51d-7d39-4751-92c5-0de8ca3880a0" (UID: "c422c51d-7d39-4751-92c5-0de8ca3880a0"). InnerVolumeSpecName "kube-api-access-55bfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:28:36.759422 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.759383 5871 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c422c51d-7d39-4751-92c5-0de8ca3880a0-dshm\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:36.759422 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.759419 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55bfp\" (UniqueName: \"kubernetes.io/projected/c422c51d-7d39-4751-92c5-0de8ca3880a0-kube-api-access-55bfp\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:36.759422 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:36.759431 5871 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c422c51d-7d39-4751-92c5-0de8ca3880a0-tls-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:37.028237 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:37.028124 5871 generic.go:356] "Generic (PLEG): container finished" podID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerID="56c4dabe19fcafc8494fb96e1a0aded7347241adcb50973455eef756f7f83129" exitCode=0 Nov 02 22:28:37.028729 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:37.028698 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" event={"ID":"c422c51d-7d39-4751-92c5-0de8ca3880a0","Type":"ContainerDied","Data":"56c4dabe19fcafc8494fb96e1a0aded7347241adcb50973455eef756f7f83129"} Nov 02 22:28:37.028885 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:37.028856 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" event={"ID":"c422c51d-7d39-4751-92c5-0de8ca3880a0","Type":"ContainerDied","Data":"ba93be5c11dadd7c02b888b94757ce0f074de32557f72c5775c9b6be943b858b"} Nov 02 22:28:37.029007 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:37.028995 5871 scope.go:122] "RemoveContainer" containerID="56c4dabe19fcafc8494fb96e1a0aded7347241adcb50973455eef756f7f83129" Nov 02 22:28:37.029275 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:37.029248 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr" Nov 02 22:28:37.060476 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:37.059702 5871 scope.go:122] "RemoveContainer" containerID="56c4dabe19fcafc8494fb96e1a0aded7347241adcb50973455eef756f7f83129" Nov 02 22:28:37.060895 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:28:37.060777 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c4dabe19fcafc8494fb96e1a0aded7347241adcb50973455eef756f7f83129\": container with ID starting with 56c4dabe19fcafc8494fb96e1a0aded7347241adcb50973455eef756f7f83129 not found: ID does not exist" containerID="56c4dabe19fcafc8494fb96e1a0aded7347241adcb50973455eef756f7f83129" Nov 02 22:28:37.060895 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:37.060816 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c4dabe19fcafc8494fb96e1a0aded7347241adcb50973455eef756f7f83129"} err="failed to get container status \"56c4dabe19fcafc8494fb96e1a0aded7347241adcb50973455eef756f7f83129\": rpc error: code = NotFound desc = could not find container \"56c4dabe19fcafc8494fb96e1a0aded7347241adcb50973455eef756f7f83129\": container with ID starting with 56c4dabe19fcafc8494fb96e1a0aded7347241adcb50973455eef756f7f83129 not found: ID does not exist" Nov 02 22:28:37.265811 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:37.252944 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr"] Nov 02 22:28:37.277152 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:37.277100 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-711bd4/distinct-w0-711bd4-kserve-7f5bf4d8b-6s2fr"] Nov 02 22:28:37.289133 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:37.288777 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:37.374689 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:37.374629 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" path="/var/lib/kubelet/pods/c422c51d-7d39-4751-92c5-0de8ca3880a0/volumes" Nov 02 22:28:37.529287 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:37.528383 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:37.719907 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:37.716475 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:40.379899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:40.330161 5871 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:40.379899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:40.330609 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 02 22:28:40.522986 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:40.522275 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:40.600196 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:40.600156 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:40.694994 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:40.694318 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 02 22:28:40.908898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:40.907983 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7" podUID="ccce4075-668d-487a-8a75-fbd41fee023a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:40.908898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:40.908453 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:41.057956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:41.054962 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:41.297150 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:41.295969 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Nov 02 22:28:41.306904 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:41.299987 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:41.360961 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:41.335979 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 02 22:28:41.360961 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:41.353101 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:28:41.620071 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:41.620020 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 02 22:28:41.927668 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:41.927568 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:28:41.927668 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:41.927634 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:28:41.927668 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:41.927663 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:28:41.927972 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:41.927693 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:28:43.772462 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:28:43.772421 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:28:44.176957 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:44.176916 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:44.586620 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:44.586522 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:44.793946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:44.792859 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:44.978750 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:44.978716 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:46.245347 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:46.245311 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:49.677109 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:49.677070 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:52.027587 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:52.027505 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:54.493169 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:54.492966 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:58.633567 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:58.633528 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:58.809353 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:58.809318 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:59.701151 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:59.701087 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:28:59.883618 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:28:59.883584 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:00.745882 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.745848 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj"] Nov 02 22:29:00.746565 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.746550 5871 cpu_manager.go:414] "containerMap: removing container" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" Nov 02 22:29:00.746634 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.746566 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" Nov 02 22:29:00.746674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.746658 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="c422c51d-7d39-4751-92c5-0de8ca3880a0" containerName="main" Nov 02 22:29:00.753702 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.753663 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.860057 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.860012 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dcefe6-1e67-4ba4-98a6-9358212dee30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.860235 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.860067 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dcefe6-1e67-4ba4-98a6-9358212dee30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.860235 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.860163 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dcefe6-1e67-4ba4-98a6-9358212dee30-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.860361 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.860281 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkvgx\" (UniqueName: \"kubernetes.io/projected/76dcefe6-1e67-4ba4-98a6-9358212dee30-kube-api-access-dkvgx\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.860414 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.860363 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dcefe6-1e67-4ba4-98a6-9358212dee30-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.860452 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.860428 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dcefe6-1e67-4ba4-98a6-9358212dee30-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.860511 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.860494 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dcefe6-1e67-4ba4-98a6-9358212dee30-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.860609 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.860593 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76dcefe6-1e67-4ba4-98a6-9358212dee30-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.860790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.860770 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dcefe6-1e67-4ba4-98a6-9358212dee30-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.877378 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.877331 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj"] Nov 02 22:29:00.962331 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.962287 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dcefe6-1e67-4ba4-98a6-9358212dee30-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.962521 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.962365 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dcefe6-1e67-4ba4-98a6-9358212dee30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.962521 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.962392 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dcefe6-1e67-4ba4-98a6-9358212dee30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.962521 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.962410 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dcefe6-1e67-4ba4-98a6-9358212dee30-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.962521 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.962447 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkvgx\" (UniqueName: \"kubernetes.io/projected/76dcefe6-1e67-4ba4-98a6-9358212dee30-kube-api-access-dkvgx\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.962659 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.962614 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dcefe6-1e67-4ba4-98a6-9358212dee30-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.962710 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.962667 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dcefe6-1e67-4ba4-98a6-9358212dee30-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.962766 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.962729 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dcefe6-1e67-4ba4-98a6-9358212dee30-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.962821 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.962759 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76dcefe6-1e67-4ba4-98a6-9358212dee30-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.962821 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.962781 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76dcefe6-1e67-4ba4-98a6-9358212dee30-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.963126 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.962832 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dcefe6-1e67-4ba4-98a6-9358212dee30-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.963189 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.963127 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76dcefe6-1e67-4ba4-98a6-9358212dee30-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.963337 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.963313 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76dcefe6-1e67-4ba4-98a6-9358212dee30-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.963375 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.963344 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76dcefe6-1e67-4ba4-98a6-9358212dee30-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.965040 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.965019 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76dcefe6-1e67-4ba4-98a6-9358212dee30-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:00.965307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:00.965291 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76dcefe6-1e67-4ba4-98a6-9358212dee30-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:01.024384 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:01.024305 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76dcefe6-1e67-4ba4-98a6-9358212dee30-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:01.024523 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:01.024426 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkvgx\" (UniqueName: \"kubernetes.io/projected/76dcefe6-1e67-4ba4-98a6-9358212dee30-kube-api-access-dkvgx\") pod \"maas-default-gateway-openshift-default-76f4876667-tfgpj\" (UID: \"76dcefe6-1e67-4ba4-98a6-9358212dee30\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:01.071607 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:01.071542 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:01.515344 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:01.515314 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj"] Nov 02 22:29:01.516798 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:29:01.516771 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76dcefe6_1e67_4ba4_98a6_9358212dee30.slice/crio-c782113b5340ed9d61897ce8d5bd9f3c8d7f35032e2bc95aa4a559a067092f5e WatchSource:0}: Error finding container c782113b5340ed9d61897ce8d5bd9f3c8d7f35032e2bc95aa4a559a067092f5e: Status 404 returned error can't find the container with id c782113b5340ed9d61897ce8d5bd9f3c8d7f35032e2bc95aa4a559a067092f5e Nov 02 22:29:01.519392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:01.519326 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:29:01.519485 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:01.519428 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:29:01.519485 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:01.519461 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:29:02.067556 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:02.067508 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" event={"ID":"76dcefe6-1e67-4ba4-98a6-9358212dee30","Type":"ContainerStarted","Data":"4ecd746cda88c55312b10682b98b8e275df69745c0c2c4b659e70523fdc2f54d"} Nov 02 22:29:02.068010 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:02.067583 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" event={"ID":"76dcefe6-1e67-4ba4-98a6-9358212dee30","Type":"ContainerStarted","Data":"c782113b5340ed9d61897ce8d5bd9f3c8d7f35032e2bc95aa4a559a067092f5e"} Nov 02 22:29:02.072163 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:02.072131 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:02.073975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:02.073950 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tfgpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:29:02.074094 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:02.073998 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" podUID="76dcefe6-1e67-4ba4-98a6-9358212dee30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 22:29:02.193542 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:02.193480 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" podStartSLOduration=2.193461912 podStartE2EDuration="2.193461912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:02.190308219 +0000 UTC m=+1365.002638398" watchObservedRunningTime="2026-11-02 22:29:02.193461912 +0000 UTC m=+1365.005792090" Nov 02 22:29:03.074899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:03.074354 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tfgpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:29:03.074899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:03.074433 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" podUID="76dcefe6-1e67-4ba4-98a6-9358212dee30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 22:29:03.603245 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:03.603205 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:29:04.072897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.072217 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tfgpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:29:04.072897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.072281 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" podUID="76dcefe6-1e67-4ba4-98a6-9358212dee30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 22:29:04.543037 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.542993 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk"] Nov 02 22:29:04.548602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.548568 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:04.686896 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.686501 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w4pz\" (UniqueName: \"kubernetes.io/projected/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-kube-api-access-9w4pz\") pod \"maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:04.686896 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.686569 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-metrics-certs\") pod \"maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:04.703935 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.703889 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-openshift-service-ca\") pod \"maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:04.741966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.708209 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2e86dfb1-serving-cert\"" Nov 02 22:29:04.742332 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.741469 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-maas-api-tls\") pod \"maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:04.802854 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.802763 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:04.824859 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.824823 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk"] Nov 02 22:29:04.905687 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.905646 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-openshift-service-ca\") pod \"maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:04.906126 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.906099 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-maas-api-tls\") pod \"maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:04.906899 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:29:04.906254 5871 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-serving-cert: secret "maas-api-e2e-derive-2e86dfb1-serving-cert" not found Nov 02 22:29:04.906899 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:29:04.906336 5871 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-maas-api-tls podName:7bd4c657-c3f1-424c-9d64-e821cdf2eb2a nodeName:}" failed. No retries permitted until 2026-11-02 22:29:05.406312923 +0000 UTC m=+1368.218643079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-maas-api-tls") pod "maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" (UID: "7bd4c657-c3f1-424c-9d64-e821cdf2eb2a") : secret "maas-api-e2e-derive-2e86dfb1-serving-cert" not found Nov 02 22:29:04.906899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.906470 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w4pz\" (UniqueName: \"kubernetes.io/projected/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-kube-api-access-9w4pz\") pod \"maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:04.906899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.906516 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-metrics-certs\") pod \"maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:04.907935 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.907914 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-openshift-service-ca\") pod \"maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:04.917336 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.917303 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-metrics-certs\") pod \"maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:04.986893 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.986797 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:04.987265 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:04.987234 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w4pz\" (UniqueName: \"kubernetes.io/projected/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-kube-api-access-9w4pz\") pod \"maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:05.073956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:05.073301 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tfgpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:29:05.073956 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:05.073381 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" podUID="76dcefe6-1e67-4ba4-98a6-9358212dee30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 22:29:05.176947 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:05.176911 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:05.414223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:05.414168 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-maas-api-tls\") pod \"maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:05.416943 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:05.416909 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-maas-api-tls\") pod \"maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:05.420606 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:05.420576 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:05.506965 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:05.506642 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:05.626090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:05.625981 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:06.104971 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:06.092316 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tfgpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:29:06.104971 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:06.092400 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" podUID="76dcefe6-1e67-4ba4-98a6-9358212dee30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 22:29:06.107862 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:06.107817 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk"] Nov 02 22:29:07.078900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:07.078427 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tfgpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:29:07.078900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:07.078506 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" podUID="76dcefe6-1e67-4ba4-98a6-9358212dee30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 22:29:07.244785 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:07.244742 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" event={"ID":"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a","Type":"ContainerStarted","Data":"7ade812063bd1365a9f2bfaf236a5c5ceee4114d6a6c8f28ec7039f6dc522e66"} Nov 02 22:29:08.075902 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:08.075743 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tfgpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:29:08.075902 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:08.075810 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" podUID="76dcefe6-1e67-4ba4-98a6-9358212dee30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 22:29:08.284901 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:08.275845 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" event={"ID":"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a","Type":"ContainerStarted","Data":"bd2ea7052b7c75ea5725e66bb39a56b50d906ebea6e55777f513330468203e53"} Nov 02 22:29:08.284901 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:08.284059 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:08.407974 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:08.407918 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" podStartSLOduration=3.3524817909999998 podStartE2EDuration="4.407896659s" totalImagesPullingTime="1.055414868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:08.404442631 +0000 UTC m=+1371.216772817" watchObservedRunningTime="2026-11-02 22:29:08.407896659 +0000 UTC m=+1371.220226837" Nov 02 22:29:09.073342 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:09.073302 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tfgpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:29:09.073644 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:09.073615 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" podUID="76dcefe6-1e67-4ba4-98a6-9358212dee30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 22:29:10.523574 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:10.523532 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:11.085899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:11.085388 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tfgpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:29:11.085899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:11.085462 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" podUID="76dcefe6-1e67-4ba4-98a6-9358212dee30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:29:11.086184 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:11.086075 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:11.292894 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:11.292841 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:12.087716 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:12.086913 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tfgpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:29:12.087716 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:12.086979 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" podUID="76dcefe6-1e67-4ba4-98a6-9358212dee30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:29:13.128018 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:13.127978 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tfgpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:29:13.128466 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:13.128049 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" podUID="76dcefe6-1e67-4ba4-98a6-9358212dee30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:29:13.695186 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:13.693943 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:13.695365 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:13.695341 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:13.705046 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:13.705006 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" Nov 02 22:29:13.788184 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:29:13.788131 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:29:15.317214 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.317179 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:29:15.767016 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.766983 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5"] Nov 02 22:29:15.770896 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.770853 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.804830 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.804790 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:15.826203 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.826164 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440d8a6a-030e-4aad-a190-85fc0292617b-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.826406 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.826220 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440d8a6a-030e-4aad-a190-85fc0292617b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.826406 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.826286 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440d8a6a-030e-4aad-a190-85fc0292617b-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.826406 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.826357 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpgtw\" (UniqueName: \"kubernetes.io/projected/440d8a6a-030e-4aad-a190-85fc0292617b-kube-api-access-vpgtw\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.826585 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.826459 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440d8a6a-030e-4aad-a190-85fc0292617b-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.826643 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.826577 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440d8a6a-030e-4aad-a190-85fc0292617b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.826704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.826648 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440d8a6a-030e-4aad-a190-85fc0292617b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.826754 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.826715 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440d8a6a-030e-4aad-a190-85fc0292617b-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.826854 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.826835 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440d8a6a-030e-4aad-a190-85fc0292617b-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.846898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.845842 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5"] Nov 02 22:29:15.928265 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.928228 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440d8a6a-030e-4aad-a190-85fc0292617b-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.928476 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.928285 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440d8a6a-030e-4aad-a190-85fc0292617b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.928476 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.928314 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440d8a6a-030e-4aad-a190-85fc0292617b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.928476 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.928354 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440d8a6a-030e-4aad-a190-85fc0292617b-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.928476 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.928408 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440d8a6a-030e-4aad-a190-85fc0292617b-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.928702 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.928540 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440d8a6a-030e-4aad-a190-85fc0292617b-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.928702 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.928585 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440d8a6a-030e-4aad-a190-85fc0292617b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.928702 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.928613 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440d8a6a-030e-4aad-a190-85fc0292617b-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.928855 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.928709 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpgtw\" (UniqueName: \"kubernetes.io/projected/440d8a6a-030e-4aad-a190-85fc0292617b-kube-api-access-vpgtw\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.928855 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.928795 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440d8a6a-030e-4aad-a190-85fc0292617b-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.928855 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.928820 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440d8a6a-030e-4aad-a190-85fc0292617b-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.929110 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.929089 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440d8a6a-030e-4aad-a190-85fc0292617b-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.929179 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.929160 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440d8a6a-030e-4aad-a190-85fc0292617b-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.929416 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.929395 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440d8a6a-030e-4aad-a190-85fc0292617b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.931278 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.931260 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440d8a6a-030e-4aad-a190-85fc0292617b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.931475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.931459 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440d8a6a-030e-4aad-a190-85fc0292617b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.990351 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.990277 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpgtw\" (UniqueName: \"kubernetes.io/projected/440d8a6a-030e-4aad-a190-85fc0292617b-kube-api-access-vpgtw\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:15.990351 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:15.990306 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440d8a6a-030e-4aad-a190-85fc0292617b-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-jh4w5\" (UID: \"440d8a6a-030e-4aad-a190-85fc0292617b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:16.097329 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:16.097285 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:16.412008 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:16.411969 5871 generic.go:356] "Generic (PLEG): container finished" podID="b2a975b1-786e-42dd-88fa-9bdd36a50ffe" containerID="37eeaa2c5b8da2e6bfbb7efe937577bcb38b71cc482938840bf53428c0646ce6" exitCode=0 Nov 02 22:29:16.412532 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:16.412115 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" event={"ID":"b2a975b1-786e-42dd-88fa-9bdd36a50ffe","Type":"ContainerDied","Data":"37eeaa2c5b8da2e6bfbb7efe937577bcb38b71cc482938840bf53428c0646ce6"} Nov 02 22:29:16.413563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:16.413539 5871 scope.go:122] "RemoveContainer" containerID="37eeaa2c5b8da2e6bfbb7efe937577bcb38b71cc482938840bf53428c0646ce6" Nov 02 22:29:16.415627 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:16.415592 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:29:16.415761 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:16.415676 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:29:16.415761 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:16.415719 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:29:16.591566 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:16.591534 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5"] Nov 02 22:29:16.593588 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:29:16.593557 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440d8a6a_030e_4aad_a190_85fc0292617b.slice/crio-45391ca0376499c2125277400987242fafbb19b72493ba4dda1c6d20e08fddd7 WatchSource:0}: Error finding container 45391ca0376499c2125277400987242fafbb19b72493ba4dda1c6d20e08fddd7: Status 404 returned error can't find the container with id 45391ca0376499c2125277400987242fafbb19b72493ba4dda1c6d20e08fddd7 Nov 02 22:29:16.597048 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:16.597012 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:29:16.597163 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:16.597096 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:29:16.597163 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:16.597143 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:29:17.427967 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:17.426702 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" event={"ID":"b2a975b1-786e-42dd-88fa-9bdd36a50ffe","Type":"ContainerStarted","Data":"420ae1da13c3ff5bc8f9c718225256aa39a3de25b215371f5de7b6ce07d7b09b"} Nov 02 22:29:17.432275 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:17.431348 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" event={"ID":"440d8a6a-030e-4aad-a190-85fc0292617b","Type":"ContainerStarted","Data":"a9e2bcec08dd3c72f014547c55c50938113e3aca014e0d1c65afeb07e144237b"} Nov 02 22:29:17.432275 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:17.431413 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" event={"ID":"440d8a6a-030e-4aad-a190-85fc0292617b","Type":"ContainerStarted","Data":"45391ca0376499c2125277400987242fafbb19b72493ba4dda1c6d20e08fddd7"} Nov 02 22:29:17.597900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:17.589011 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:29:17.610897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:17.610154 5871 patch_prober.go:29] interesting pod/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 22:29:17.610897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:17.610317 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" podUID="b2a975b1-786e-42dd-88fa-9bdd36a50ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 22:29:17.714955 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:17.714442 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" podStartSLOduration=2.714423151 podStartE2EDuration="2.714423151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:17.712311661 +0000 UTC m=+1380.524641840" watchObservedRunningTime="2026-11-02 22:29:17.714423151 +0000 UTC m=+1380.526753331" Nov 02 22:29:18.102962 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:18.102126 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:18.104599 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:18.104516 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jh4w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 22:29:18.104599 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:18.104574 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" podUID="440d8a6a-030e-4aad-a190-85fc0292617b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 22:29:18.593900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:18.590288 5871 patch_prober.go:29] interesting pod/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 22:29:18.593900 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:18.590357 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" podUID="b2a975b1-786e-42dd-88fa-9bdd36a50ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 22:29:19.098897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:19.098325 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jh4w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 22:29:19.098897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:19.098397 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" podUID="440d8a6a-030e-4aad-a190-85fc0292617b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 22:29:19.594897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:19.594333 5871 patch_prober.go:29] interesting pod/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 22:29:19.594897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:19.594400 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" podUID="b2a975b1-786e-42dd-88fa-9bdd36a50ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 22:29:20.104414 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:20.104371 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jh4w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 22:29:20.104744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:20.104713 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" podUID="440d8a6a-030e-4aad-a190-85fc0292617b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 22:29:20.590242 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:20.590204 5871 patch_prober.go:29] interesting pod/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 22:29:20.590552 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:20.590524 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" podUID="b2a975b1-786e-42dd-88fa-9bdd36a50ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 22:29:20.874776 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:20.874739 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:21.069446 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:21.069409 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:21.108935 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:21.108902 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jh4w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 22:29:21.119054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:21.119009 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" podUID="440d8a6a-030e-4aad-a190-85fc0292617b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 22:29:21.590953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:21.590913 5871 patch_prober.go:29] interesting pod/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 22:29:21.591263 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:21.591232 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" podUID="b2a975b1-786e-42dd-88fa-9bdd36a50ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 22:29:22.100920 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:22.100305 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jh4w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 22:29:22.100920 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:22.100371 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" podUID="440d8a6a-030e-4aad-a190-85fc0292617b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 22:29:22.477911 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:22.476979 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:29:22.590347 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:22.590304 5871 patch_prober.go:29] interesting pod/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 22:29:22.590635 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:22.590601 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" podUID="b2a975b1-786e-42dd-88fa-9bdd36a50ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 22:29:23.098243 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:23.098178 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jh4w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 22:29:23.098445 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:23.098281 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" podUID="440d8a6a-030e-4aad-a190-85fc0292617b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 22:29:23.593102 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:23.593066 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:29:23.594434 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:23.594409 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:29:24.097704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:24.097666 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jh4w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 22:29:24.097933 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:24.097727 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" podUID="440d8a6a-030e-4aad-a190-85fc0292617b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 22:29:25.735521 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:25.735486 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:25.736459 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:25.736426 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:25.738034 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:25.738015 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" Nov 02 22:29:26.432987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:26.432939 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:27.826298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:27.826212 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:28.014310 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:28.014275 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:31.873293 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:31.873258 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:37.259897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:37.259545 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:40.825192 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:40.825147 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:42.571487 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:42.571447 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:43.077351 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:43.077312 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:43.308715 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:43.308679 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:43.771312 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:29:43.771266 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:29:44.556606 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:44.556574 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:47.912445 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:47.912386 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:48.593116 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:48.593062 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:48.771638 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:48.771606 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:50.941415 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:50.941374 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:51.123396 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:51.123359 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:53.257707 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:53.257669 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:57.454124 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:57.454086 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:29:58.939439 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:58.939402 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:29:59.043535 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:29:59.043482 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:00.193656 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:00.193618 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6"] Nov 02 22:30:00.197829 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:00.197804 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" Nov 02 22:30:00.273006 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:00.272964 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfjcb\" (UniqueName: \"kubernetes.io/projected/23fb37b7-5daf-4ef7-a553-3037abfe4a81-kube-api-access-hfjcb\") pod \"maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6\" (UID: \"23fb37b7-5daf-4ef7-a553-3037abfe4a81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" Nov 02 22:30:00.273299 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:00.273276 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23fb37b7-5daf-4ef7-a553-3037abfe4a81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6\" (UID: \"23fb37b7-5daf-4ef7-a553-3037abfe4a81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" Nov 02 22:30:00.275841 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:00.275813 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6"] Nov 02 22:30:00.360945 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:00.360908 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cndr2\"" Nov 02 22:30:00.374798 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:00.374762 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfjcb\" (UniqueName: \"kubernetes.io/projected/23fb37b7-5daf-4ef7-a553-3037abfe4a81-kube-api-access-hfjcb\") pod \"maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6\" (UID: \"23fb37b7-5daf-4ef7-a553-3037abfe4a81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" Nov 02 22:30:00.375018 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:00.374827 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23fb37b7-5daf-4ef7-a553-3037abfe4a81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6\" (UID: \"23fb37b7-5daf-4ef7-a553-3037abfe4a81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" Nov 02 22:30:00.375534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:00.375513 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23fb37b7-5daf-4ef7-a553-3037abfe4a81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6\" (UID: \"23fb37b7-5daf-4ef7-a553-3037abfe4a81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" Nov 02 22:30:00.436629 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:00.436591 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfjcb\" (UniqueName: \"kubernetes.io/projected/23fb37b7-5daf-4ef7-a553-3037abfe4a81-kube-api-access-hfjcb\") pod \"maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6\" (UID: \"23fb37b7-5daf-4ef7-a553-3037abfe4a81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" Nov 02 22:30:00.512401 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:00.512317 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" Nov 02 22:30:00.654110 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:00.654067 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:00.897070 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:00.897041 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6"] Nov 02 22:30:00.899240 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:30:00.899209 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23fb37b7_5daf_4ef7_a553_3037abfe4a81.slice/crio-710e96ce2c489525ffc67d94d9e514ba59af76e0ab2e25e5424543c542d448e8 WatchSource:0}: Error finding container 710e96ce2c489525ffc67d94d9e514ba59af76e0ab2e25e5424543c542d448e8: Status 404 returned error can't find the container with id 710e96ce2c489525ffc67d94d9e514ba59af76e0ab2e25e5424543c542d448e8 Nov 02 22:30:01.829729 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:01.829690 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" event={"ID":"23fb37b7-5daf-4ef7-a553-3037abfe4a81","Type":"ContainerStarted","Data":"710e96ce2c489525ffc67d94d9e514ba59af76e0ab2e25e5424543c542d448e8"} Nov 02 22:30:04.876543 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:04.876476 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:05.800339 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:05.800299 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:06.690097 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:06.690059 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:06.875336 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:06.875293 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:07.014866 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:07.014838 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck"] Nov 02 22:30:07.015114 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:07.015087 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" podUID="2e9d9f83-67f8-44dd-89a6-1c622fd4708a" containerName="istio-proxy" containerID="cri-o://3f3581db6a9bcbb3fa4a17d62a72b29243cd76f271540e47b13655f5f3e29936" gracePeriod=30 Nov 02 22:30:07.029790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:07.029762 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z"] Nov 02 22:30:07.030041 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:07.030013 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" podUID="9b8e2845-337e-4fa5-b745-ae219273156b" containerName="istio-proxy" containerID="cri-o://bf95e4eb6003ab689024a0bfebb1fbac9484c0c47ed01b17e21a079f5e5516cd" gracePeriod=30 Nov 02 22:30:07.038670 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:07.038646 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br"] Nov 02 22:30:07.038845 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:07.038826 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" podUID="c910d204-821e-4a39-9f51-40668aaa2e7e" containerName="istio-proxy" containerID="cri-o://04685e926389071813a0c82cea20a192bd9671faa5e5fdd67091038fd6668b18" gracePeriod=30 Nov 02 22:30:07.058571 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:07.058538 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:07.901441 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:07.901398 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" event={"ID":"23fb37b7-5daf-4ef7-a553-3037abfe4a81","Type":"ContainerStarted","Data":"3861d2243a99312d5e7dcdcdde079832e2051891ee8e13ce919ac7f4bb43cd04"} Nov 02 22:30:08.021011 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:08.020948 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" podStartSLOduration=1.914771582 podStartE2EDuration="8.020928734s" totalImagesPullingTime="6.106157152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:08.017369198 +0000 UTC m=+1430.829699385" watchObservedRunningTime="2026-11-02 22:30:08.020928734 +0000 UTC m=+1430.833258917" Nov 02 22:30:08.910810 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:08.910771 5871 generic.go:356] "Generic (PLEG): container finished" podID="23fb37b7-5daf-4ef7-a553-3037abfe4a81" containerID="3861d2243a99312d5e7dcdcdde079832e2051891ee8e13ce919ac7f4bb43cd04" exitCode=60 Nov 02 22:30:08.911300 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:08.910852 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" event={"ID":"23fb37b7-5daf-4ef7-a553-3037abfe4a81","Type":"ContainerDied","Data":"3861d2243a99312d5e7dcdcdde079832e2051891ee8e13ce919ac7f4bb43cd04"} Nov 02 22:30:08.911300 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:08.911178 5871 scope.go:122] "RemoveContainer" containerID="3861d2243a99312d5e7dcdcdde079832e2051891ee8e13ce919ac7f4bb43cd04" Nov 02 22:30:09.583719 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:09.583619 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:09.878750 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:09.878713 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:09.920122 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:09.920085 5871 generic.go:356] "Generic (PLEG): container finished" podID="23fb37b7-5daf-4ef7-a553-3037abfe4a81" containerID="a4eb2a61466b6383d1feab507ec5216866b381a60c59b2d69ff8f7dfa719c54c" exitCode=60 Nov 02 22:30:09.920564 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:09.920185 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" event={"ID":"23fb37b7-5daf-4ef7-a553-3037abfe4a81","Type":"ContainerDied","Data":"a4eb2a61466b6383d1feab507ec5216866b381a60c59b2d69ff8f7dfa719c54c"} Nov 02 22:30:09.920564 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:09.920252 5871 scope.go:122] "RemoveContainer" containerID="3861d2243a99312d5e7dcdcdde079832e2051891ee8e13ce919ac7f4bb43cd04" Nov 02 22:30:09.920688 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:09.920634 5871 scope.go:122] "RemoveContainer" containerID="a4eb2a61466b6383d1feab507ec5216866b381a60c59b2d69ff8f7dfa719c54c" Nov 02 22:30:09.921157 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:30:09.921120 5871 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6_odh-ai-gateway-infra(23fb37b7-5daf-4ef7-a553-3037abfe4a81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" podUID="23fb37b7-5daf-4ef7-a553-3037abfe4a81" Nov 02 22:30:10.105075 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:10.105034 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:10.940719 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:10.940687 5871 scope.go:122] "RemoveContainer" containerID="a4eb2a61466b6383d1feab507ec5216866b381a60c59b2d69ff8f7dfa719c54c" Nov 02 22:30:10.941230 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:30:10.941055 5871 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6_odh-ai-gateway-infra(23fb37b7-5daf-4ef7-a553-3037abfe4a81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" podUID="23fb37b7-5daf-4ef7-a553-3037abfe4a81" Nov 02 22:30:12.339055 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.339018 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:30:12.375987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.371392 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-data\") pod \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " Nov 02 22:30:12.375987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.371465 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-kube-api-access-xqjtx\" (UniqueName: \"kubernetes.io/projected/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-kube-api-access-xqjtx\") pod \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " Nov 02 22:30:12.375987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.371501 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-workload-certs\") pod \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " Nov 02 22:30:12.375987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.371805 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-data" pod "2e9d9f83-67f8-44dd-89a6-1c622fd4708a" (UID: "2e9d9f83-67f8-44dd-89a6-1c622fd4708a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.375987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.372981 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-credential-socket\") pod \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " Nov 02 22:30:12.375987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.373037 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-envoy\") pod \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " Nov 02 22:30:12.375987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.373075 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-token\" (UniqueName: \"kubernetes.io/projected/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-token\") pod \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " Nov 02 22:30:12.375987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.373207 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istiod-ca-cert\") pod \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " Nov 02 22:30:12.375987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.373269 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-workload-socket\") pod \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " Nov 02 22:30:12.375987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.373447 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-podinfo\") pod \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\" (UID: \"2e9d9f83-67f8-44dd-89a6-1c622fd4708a\") " Nov 02 22:30:12.375987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.374226 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.375987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.374380 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istiod-ca-cert" pod "2e9d9f83-67f8-44dd-89a6-1c622fd4708a" (UID: "2e9d9f83-67f8-44dd-89a6-1c622fd4708a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:12.375987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.374781 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-workload-certs" pod "2e9d9f83-67f8-44dd-89a6-1c622fd4708a" (UID: "2e9d9f83-67f8-44dd-89a6-1c622fd4708a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.375987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.375096 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-workload-socket" pod "2e9d9f83-67f8-44dd-89a6-1c622fd4708a" (UID: "2e9d9f83-67f8-44dd-89a6-1c622fd4708a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.375987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.375267 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-credential-socket" pod "2e9d9f83-67f8-44dd-89a6-1c622fd4708a" (UID: "2e9d9f83-67f8-44dd-89a6-1c622fd4708a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.377712 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.377679 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-envoy" pod "2e9d9f83-67f8-44dd-89a6-1c622fd4708a" (UID: "2e9d9f83-67f8-44dd-89a6-1c622fd4708a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.378145 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.378118 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-kube-api-access-xqjtx" pod "2e9d9f83-67f8-44dd-89a6-1c622fd4708a" (UID: "2e9d9f83-67f8-44dd-89a6-1c622fd4708a"). InnerVolumeSpecName "kube-api-access-xqjtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:12.378478 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.378453 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-podinfo" pod "2e9d9f83-67f8-44dd-89a6-1c622fd4708a" (UID: "2e9d9f83-67f8-44dd-89a6-1c622fd4708a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:30:12.380184 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.380153 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-token" pod "2e9d9f83-67f8-44dd-89a6-1c622fd4708a" (UID: "2e9d9f83-67f8-44dd-89a6-1c622fd4708a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:12.403500 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.403474 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:30:12.408154 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.408125 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:30:12.475717 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.475682 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-workload-certs\") pod \"9b8e2845-337e-4fa5-b745-ae219273156b\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " Nov 02 22:30:12.475940 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.475740 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-credential-socket\") pod \"c910d204-821e-4a39-9f51-40668aaa2e7e\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " Nov 02 22:30:12.475940 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.475783 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-workload-socket\") pod \"9b8e2845-337e-4fa5-b745-ae219273156b\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " Nov 02 22:30:12.475940 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.475811 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-istio-data\") pod \"9b8e2845-337e-4fa5-b745-ae219273156b\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " Nov 02 22:30:12.475940 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.475914 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-token\" (UniqueName: \"kubernetes.io/projected/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-token\") pod \"c910d204-821e-4a39-9f51-40668aaa2e7e\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " Nov 02 22:30:12.476178 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.475954 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b8e2845-337e-4fa5-b745-ae219273156b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8e2845-337e-4fa5-b745-ae219273156b-istiod-ca-cert\") pod \"9b8e2845-337e-4fa5-b745-ae219273156b\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " Nov 02 22:30:12.476178 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476016 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b8e2845-337e-4fa5-b745-ae219273156b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8e2845-337e-4fa5-b745-ae219273156b-istio-podinfo\") pod \"9b8e2845-337e-4fa5-b745-ae219273156b\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " Nov 02 22:30:12.476178 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476030 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-credential-socket" pod "c910d204-821e-4a39-9f51-40668aaa2e7e" (UID: "c910d204-821e-4a39-9f51-40668aaa2e7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.476178 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476052 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-workload-certs" pod "9b8e2845-337e-4fa5-b745-ae219273156b" (UID: "9b8e2845-337e-4fa5-b745-ae219273156b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.476178 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476096 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-workload-socket\") pod \"c910d204-821e-4a39-9f51-40668aaa2e7e\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " Nov 02 22:30:12.476430 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476179 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-workload-socket" pod "9b8e2845-337e-4fa5-b745-ae219273156b" (UID: "9b8e2845-337e-4fa5-b745-ae219273156b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.476430 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476205 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-istio-data" pod "9b8e2845-337e-4fa5-b745-ae219273156b" (UID: "9b8e2845-337e-4fa5-b745-ae219273156b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.476430 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476261 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8e2845-337e-4fa5-b745-ae219273156b-istio-token\" (UniqueName: \"kubernetes.io/projected/9b8e2845-337e-4fa5-b745-ae219273156b-istio-token\") pod \"9b8e2845-337e-4fa5-b745-ae219273156b\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " Nov 02 22:30:12.476430 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476329 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-istio-envoy\") pod \"9b8e2845-337e-4fa5-b745-ae219273156b\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " Nov 02 22:30:12.476430 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476357 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-workload-socket" pod "c910d204-821e-4a39-9f51-40668aaa2e7e" (UID: "c910d204-821e-4a39-9f51-40668aaa2e7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.476430 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476363 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-workload-certs\") pod \"c910d204-821e-4a39-9f51-40668aaa2e7e\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " Nov 02 22:30:12.476430 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476378 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8e2845-337e-4fa5-b745-ae219273156b-istiod-ca-cert" pod "9b8e2845-337e-4fa5-b745-ae219273156b" (UID: "9b8e2845-337e-4fa5-b745-ae219273156b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:12.476430 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476405 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c910d204-821e-4a39-9f51-40668aaa2e7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c910d204-821e-4a39-9f51-40668aaa2e7e-istiod-ca-cert\") pod \"c910d204-821e-4a39-9f51-40668aaa2e7e\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " Nov 02 22:30:12.476860 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476451 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-data\") pod \"c910d204-821e-4a39-9f51-40668aaa2e7e\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " Nov 02 22:30:12.476860 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476480 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-podinfo\") pod \"c910d204-821e-4a39-9f51-40668aaa2e7e\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " Nov 02 22:30:12.476860 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476533 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c910d204-821e-4a39-9f51-40668aaa2e7e-kube-api-access-t5hrk\" (UniqueName: \"kubernetes.io/projected/c910d204-821e-4a39-9f51-40668aaa2e7e-kube-api-access-t5hrk\") pod \"c910d204-821e-4a39-9f51-40668aaa2e7e\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " Nov 02 22:30:12.476860 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476684 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-credential-socket\") pod \"9b8e2845-337e-4fa5-b745-ae219273156b\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " Nov 02 22:30:12.476860 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476750 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-workload-certs" pod "c910d204-821e-4a39-9f51-40668aaa2e7e" (UID: "c910d204-821e-4a39-9f51-40668aaa2e7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.476860 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476772 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-envoy\") pod \"c910d204-821e-4a39-9f51-40668aaa2e7e\" (UID: \"c910d204-821e-4a39-9f51-40668aaa2e7e\") " Nov 02 22:30:12.476860 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476804 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8e2845-337e-4fa5-b745-ae219273156b-kube-api-access-t6tmx\" (UniqueName: \"kubernetes.io/projected/9b8e2845-337e-4fa5-b745-ae219273156b-kube-api-access-t6tmx\") pod \"9b8e2845-337e-4fa5-b745-ae219273156b\" (UID: \"9b8e2845-337e-4fa5-b745-ae219273156b\") " Nov 02 22:30:12.476860 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.476806 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c910d204-821e-4a39-9f51-40668aaa2e7e-istiod-ca-cert" pod "c910d204-821e-4a39-9f51-40668aaa2e7e" (UID: "c910d204-821e-4a39-9f51-40668aaa2e7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:12.477270 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477238 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-data" pod "c910d204-821e-4a39-9f51-40668aaa2e7e" (UID: "c910d204-821e-4a39-9f51-40668aaa2e7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.477556 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477530 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-credential-socket" pod "9b8e2845-337e-4fa5-b745-ae219273156b" (UID: "9b8e2845-337e-4fa5-b745-ae219273156b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477720 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477748 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqjtx\" (UniqueName: \"kubernetes.io/projected/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-kube-api-access-xqjtx\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477765 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477779 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477792 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477807 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477822 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477836 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477850 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477864 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477898 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8e2845-337e-4fa5-b745-ae219273156b-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477913 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477927 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477943 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477957 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e9d9f83-67f8-44dd-89a6-1c622fd4708a-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.477989 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.478307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.478004 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c910d204-821e-4a39-9f51-40668aaa2e7e-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.479189 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.478868 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-istio-envoy" pod "9b8e2845-337e-4fa5-b745-ae219273156b" (UID: "9b8e2845-337e-4fa5-b745-ae219273156b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.479189 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.478867 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8e2845-337e-4fa5-b745-ae219273156b-istio-token" pod "9b8e2845-337e-4fa5-b745-ae219273156b" (UID: "9b8e2845-337e-4fa5-b745-ae219273156b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:12.479189 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.479149 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b8e2845-337e-4fa5-b745-ae219273156b-istio-podinfo" pod "9b8e2845-337e-4fa5-b745-ae219273156b" (UID: "9b8e2845-337e-4fa5-b745-ae219273156b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:30:12.479358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.479199 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-token" pod "c910d204-821e-4a39-9f51-40668aaa2e7e" (UID: "c910d204-821e-4a39-9f51-40668aaa2e7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:12.479608 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.479588 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8e2845-337e-4fa5-b745-ae219273156b-kube-api-access-t6tmx" pod "9b8e2845-337e-4fa5-b745-ae219273156b" (UID: "9b8e2845-337e-4fa5-b745-ae219273156b"). InnerVolumeSpecName "kube-api-access-t6tmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:12.479939 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.479921 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c910d204-821e-4a39-9f51-40668aaa2e7e-kube-api-access-t5hrk" pod "c910d204-821e-4a39-9f51-40668aaa2e7e" (UID: "c910d204-821e-4a39-9f51-40668aaa2e7e"). InnerVolumeSpecName "kube-api-access-t5hrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:12.480861 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.480838 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-podinfo" pod "c910d204-821e-4a39-9f51-40668aaa2e7e" (UID: "c910d204-821e-4a39-9f51-40668aaa2e7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:30:12.481225 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.481208 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-envoy" pod "c910d204-821e-4a39-9f51-40668aaa2e7e" (UID: "c910d204-821e-4a39-9f51-40668aaa2e7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:12.579953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.579852 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.579953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.579907 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8e2845-337e-4fa5-b745-ae219273156b-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.579953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.579924 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8e2845-337e-4fa5-b745-ae219273156b-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.579953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.579939 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.579953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.579953 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.580224 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.579970 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5hrk\" (UniqueName: \"kubernetes.io/projected/c910d204-821e-4a39-9f51-40668aaa2e7e-kube-api-access-t5hrk\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.580224 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.579984 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8e2845-337e-4fa5-b745-ae219273156b-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.580224 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.579998 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c910d204-821e-4a39-9f51-40668aaa2e7e-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.580224 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.580012 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6tmx\" (UniqueName: \"kubernetes.io/projected/9b8e2845-337e-4fa5-b745-ae219273156b-kube-api-access-t6tmx\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:12.958830 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.958793 5871 generic.go:356] "Generic (PLEG): container finished" podID="9b8e2845-337e-4fa5-b745-ae219273156b" containerID="bf95e4eb6003ab689024a0bfebb1fbac9484c0c47ed01b17e21a079f5e5516cd" exitCode=0 Nov 02 22:30:12.959052 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.958882 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" Nov 02 22:30:12.959052 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.958910 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" event={"ID":"9b8e2845-337e-4fa5-b745-ae219273156b","Type":"ContainerDied","Data":"bf95e4eb6003ab689024a0bfebb1fbac9484c0c47ed01b17e21a079f5e5516cd"} Nov 02 22:30:12.959190 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.959071 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z" event={"ID":"9b8e2845-337e-4fa5-b745-ae219273156b","Type":"ContainerDied","Data":"c15ca5bf4496ea74b52d73d19cb87b8bd3b1a11727c2ea1287399b6d02145a3c"} Nov 02 22:30:12.959190 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.959112 5871 scope.go:122] "RemoveContainer" containerID="bf95e4eb6003ab689024a0bfebb1fbac9484c0c47ed01b17e21a079f5e5516cd" Nov 02 22:30:12.961671 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.961638 5871 generic.go:356] "Generic (PLEG): container finished" podID="c910d204-821e-4a39-9f51-40668aaa2e7e" containerID="04685e926389071813a0c82cea20a192bd9671faa5e5fdd67091038fd6668b18" exitCode=0 Nov 02 22:30:12.961796 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.961668 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" event={"ID":"c910d204-821e-4a39-9f51-40668aaa2e7e","Type":"ContainerDied","Data":"04685e926389071813a0c82cea20a192bd9671faa5e5fdd67091038fd6668b18"} Nov 02 22:30:12.961796 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.961711 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" Nov 02 22:30:12.961796 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.961748 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br" event={"ID":"c910d204-821e-4a39-9f51-40668aaa2e7e","Type":"ContainerDied","Data":"c4f8dc3e359a9d1d9fb3ba089f9569bb6d1430595d94b1259a8ed0d3b0db0214"} Nov 02 22:30:12.963413 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.963393 5871 generic.go:356] "Generic (PLEG): container finished" podID="2e9d9f83-67f8-44dd-89a6-1c622fd4708a" containerID="3f3581db6a9bcbb3fa4a17d62a72b29243cd76f271540e47b13655f5f3e29936" exitCode=0 Nov 02 22:30:12.963530 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.963422 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" event={"ID":"2e9d9f83-67f8-44dd-89a6-1c622fd4708a","Type":"ContainerDied","Data":"3f3581db6a9bcbb3fa4a17d62a72b29243cd76f271540e47b13655f5f3e29936"} Nov 02 22:30:12.963530 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.963452 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" Nov 02 22:30:12.963530 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.963506 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck" event={"ID":"2e9d9f83-67f8-44dd-89a6-1c622fd4708a","Type":"ContainerDied","Data":"68b52a14f9633c7b3704e7e5eba20a524b858a63b5f7929b4b24466169a8a5df"} Nov 02 22:30:12.973470 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.973374 5871 scope.go:122] "RemoveContainer" containerID="bf95e4eb6003ab689024a0bfebb1fbac9484c0c47ed01b17e21a079f5e5516cd" Nov 02 22:30:12.973791 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:30:12.973768 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf95e4eb6003ab689024a0bfebb1fbac9484c0c47ed01b17e21a079f5e5516cd\": container with ID starting with bf95e4eb6003ab689024a0bfebb1fbac9484c0c47ed01b17e21a079f5e5516cd not found: ID does not exist" containerID="bf95e4eb6003ab689024a0bfebb1fbac9484c0c47ed01b17e21a079f5e5516cd" Nov 02 22:30:12.973862 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.973803 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf95e4eb6003ab689024a0bfebb1fbac9484c0c47ed01b17e21a079f5e5516cd"} err="failed to get container status \"bf95e4eb6003ab689024a0bfebb1fbac9484c0c47ed01b17e21a079f5e5516cd\": rpc error: code = NotFound desc = could not find container \"bf95e4eb6003ab689024a0bfebb1fbac9484c0c47ed01b17e21a079f5e5516cd\": container with ID starting with bf95e4eb6003ab689024a0bfebb1fbac9484c0c47ed01b17e21a079f5e5516cd not found: ID does not exist" Nov 02 22:30:12.973862 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.973823 5871 scope.go:122] "RemoveContainer" containerID="04685e926389071813a0c82cea20a192bd9671faa5e5fdd67091038fd6668b18" Nov 02 22:30:12.984509 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.984488 5871 scope.go:122] "RemoveContainer" containerID="04685e926389071813a0c82cea20a192bd9671faa5e5fdd67091038fd6668b18" Nov 02 22:30:12.984840 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:30:12.984819 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04685e926389071813a0c82cea20a192bd9671faa5e5fdd67091038fd6668b18\": container with ID starting with 04685e926389071813a0c82cea20a192bd9671faa5e5fdd67091038fd6668b18 not found: ID does not exist" containerID="04685e926389071813a0c82cea20a192bd9671faa5e5fdd67091038fd6668b18" Nov 02 22:30:12.984942 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.984857 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04685e926389071813a0c82cea20a192bd9671faa5e5fdd67091038fd6668b18"} err="failed to get container status \"04685e926389071813a0c82cea20a192bd9671faa5e5fdd67091038fd6668b18\": rpc error: code = NotFound desc = could not find container \"04685e926389071813a0c82cea20a192bd9671faa5e5fdd67091038fd6668b18\": container with ID starting with 04685e926389071813a0c82cea20a192bd9671faa5e5fdd67091038fd6668b18 not found: ID does not exist" Nov 02 22:30:12.984942 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.984907 5871 scope.go:122] "RemoveContainer" containerID="3f3581db6a9bcbb3fa4a17d62a72b29243cd76f271540e47b13655f5f3e29936" Nov 02 22:30:12.994926 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.994898 5871 scope.go:122] "RemoveContainer" containerID="3f3581db6a9bcbb3fa4a17d62a72b29243cd76f271540e47b13655f5f3e29936" Nov 02 22:30:12.995247 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:30:12.995224 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3581db6a9bcbb3fa4a17d62a72b29243cd76f271540e47b13655f5f3e29936\": container with ID starting with 3f3581db6a9bcbb3fa4a17d62a72b29243cd76f271540e47b13655f5f3e29936 not found: ID does not exist" containerID="3f3581db6a9bcbb3fa4a17d62a72b29243cd76f271540e47b13655f5f3e29936" Nov 02 22:30:12.995332 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:12.995255 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3581db6a9bcbb3fa4a17d62a72b29243cd76f271540e47b13655f5f3e29936"} err="failed to get container status \"3f3581db6a9bcbb3fa4a17d62a72b29243cd76f271540e47b13655f5f3e29936\": rpc error: code = NotFound desc = could not find container \"3f3581db6a9bcbb3fa4a17d62a72b29243cd76f271540e47b13655f5f3e29936\": container with ID starting with 3f3581db6a9bcbb3fa4a17d62a72b29243cd76f271540e47b13655f5f3e29936 not found: ID does not exist" Nov 02 22:30:13.130797 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:13.130758 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br"] Nov 02 22:30:13.135090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:13.135054 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-vd2br"] Nov 02 22:30:13.246431 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:13.246325 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck"] Nov 02 22:30:13.250516 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:13.250484 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-n6pck"] Nov 02 22:30:13.316869 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:13.316832 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e9d9f83-67f8-44dd-89a6-1c622fd4708a" path="/var/lib/kubelet/pods/2e9d9f83-67f8-44dd-89a6-1c622fd4708a/volumes" Nov 02 22:30:13.317453 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:13.317437 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c910d204-821e-4a39-9f51-40668aaa2e7e" path="/var/lib/kubelet/pods/c910d204-821e-4a39-9f51-40668aaa2e7e/volumes" Nov 02 22:30:13.361383 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:13.361344 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z"] Nov 02 22:30:13.365639 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:13.365604 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-711bd4-openshift-default-5d96df6c75-qxc5z"] Nov 02 22:30:13.772373 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:30:13.772323 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:30:14.357289 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:14.357255 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:15.315463 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:15.315430 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b8e2845-337e-4fa5-b745-ae219273156b" path="/var/lib/kubelet/pods/9b8e2845-337e-4fa5-b745-ae219273156b/volumes" Nov 02 22:30:15.888460 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:15.888401 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:16.058214 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:16.058176 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:16.240362 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:16.240265 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:16.419881 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:16.419838 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:16.601576 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:16.601483 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:16.802981 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:16.802942 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb"] Nov 02 22:30:16.803218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:16.803174 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" podUID="b2a975b1-786e-42dd-88fa-9bdd36a50ffe" containerName="istio-proxy" containerID="cri-o://420ae1da13c3ff5bc8f9c718225256aa39a3de25b215371f5de7b6ce07d7b09b" gracePeriod=30 Nov 02 22:30:16.847407 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:16.847338 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:17.258450 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:17.258402 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:17.511997 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:17.511912 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:17.711337 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:17.711301 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:17.899161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:17.899118 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:18.668888 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:18.668817 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:18.865375 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:18.865325 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:19.322859 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:19.322825 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:21.728529 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:21.728493 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:21.767139 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:21.767107 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6"] Nov 02 22:30:22.022971 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.017415 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" Nov 02 22:30:22.049893 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.049850 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" event={"ID":"23fb37b7-5daf-4ef7-a553-3037abfe4a81","Type":"ContainerDied","Data":"710e96ce2c489525ffc67d94d9e514ba59af76e0ab2e25e5424543c542d448e8"} Nov 02 22:30:22.050142 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.049960 5871 scope.go:122] "RemoveContainer" containerID="a4eb2a61466b6383d1feab507ec5216866b381a60c59b2d69ff8f7dfa719c54c" Nov 02 22:30:22.050142 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.050106 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6" Nov 02 22:30:22.052935 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.052854 5871 generic.go:356] "Generic (PLEG): container finished" podID="b2a975b1-786e-42dd-88fa-9bdd36a50ffe" containerID="420ae1da13c3ff5bc8f9c718225256aa39a3de25b215371f5de7b6ce07d7b09b" exitCode=0 Nov 02 22:30:22.053137 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.053115 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" event={"ID":"b2a975b1-786e-42dd-88fa-9bdd36a50ffe","Type":"ContainerDied","Data":"420ae1da13c3ff5bc8f9c718225256aa39a3de25b215371f5de7b6ce07d7b09b"} Nov 02 22:30:22.069318 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.069292 5871 scope.go:122] "RemoveContainer" containerID="37eeaa2c5b8da2e6bfbb7efe937577bcb38b71cc482938840bf53428c0646ce6" Nov 02 22:30:22.131310 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.131262 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23fb37b7-5daf-4ef7-a553-3037abfe4a81-kube-api-access-hfjcb\" (UniqueName: \"kubernetes.io/projected/23fb37b7-5daf-4ef7-a553-3037abfe4a81-kube-api-access-hfjcb\") pod \"23fb37b7-5daf-4ef7-a553-3037abfe4a81\" (UID: \"23fb37b7-5daf-4ef7-a553-3037abfe4a81\") " Nov 02 22:30:22.131460 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.131393 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23fb37b7-5daf-4ef7-a553-3037abfe4a81-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23fb37b7-5daf-4ef7-a553-3037abfe4a81-maas-api-service-ca\") pod \"23fb37b7-5daf-4ef7-a553-3037abfe4a81\" (UID: \"23fb37b7-5daf-4ef7-a553-3037abfe4a81\") " Nov 02 22:30:22.132155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.132126 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23fb37b7-5daf-4ef7-a553-3037abfe4a81-maas-api-service-ca" pod "23fb37b7-5daf-4ef7-a553-3037abfe4a81" (UID: "23fb37b7-5daf-4ef7-a553-3037abfe4a81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:22.148903 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.140102 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fb37b7-5daf-4ef7-a553-3037abfe4a81-kube-api-access-hfjcb" pod "23fb37b7-5daf-4ef7-a553-3037abfe4a81" (UID: "23fb37b7-5daf-4ef7-a553-3037abfe4a81"). InnerVolumeSpecName "kube-api-access-hfjcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:22.220899 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.220214 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:30:22.240991 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.238789 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfjcb\" (UniqueName: \"kubernetes.io/projected/23fb37b7-5daf-4ef7-a553-3037abfe4a81-kube-api-access-hfjcb\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:22.240991 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.238826 5871 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23fb37b7-5daf-4ef7-a553-3037abfe4a81-maas-api-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:22.339561 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.339522 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istiod-ca-cert\") pod \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " Nov 02 22:30:22.339758 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.339645 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-token\" (UniqueName: \"kubernetes.io/projected/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-token\") pod \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " Nov 02 22:30:22.339758 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.339692 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-workload-socket\") pod \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " Nov 02 22:30:22.339758 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.339751 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-credential-socket\") pod \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " Nov 02 22:30:22.339988 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.339814 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-workload-certs\") pod \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " Nov 02 22:30:22.339988 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.339945 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-kube-api-access-8v24s\" (UniqueName: \"kubernetes.io/projected/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-kube-api-access-8v24s\") pod \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " Nov 02 22:30:22.339988 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.339978 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-envoy\") pod \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " Nov 02 22:30:22.340167 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.340149 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-data\") pod \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " Nov 02 22:30:22.340233 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.340219 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-podinfo\") pod \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\" (UID: \"b2a975b1-786e-42dd-88fa-9bdd36a50ffe\") " Nov 02 22:30:22.341609 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.341573 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-workload-certs" pod "b2a975b1-786e-42dd-88fa-9bdd36a50ffe" (UID: "b2a975b1-786e-42dd-88fa-9bdd36a50ffe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:22.342684 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.342640 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-workload-socket" pod "b2a975b1-786e-42dd-88fa-9bdd36a50ffe" (UID: "b2a975b1-786e-42dd-88fa-9bdd36a50ffe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:22.342684 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.342663 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-data" pod "b2a975b1-786e-42dd-88fa-9bdd36a50ffe" (UID: "b2a975b1-786e-42dd-88fa-9bdd36a50ffe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:22.342912 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.342732 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-credential-socket" pod "b2a975b1-786e-42dd-88fa-9bdd36a50ffe" (UID: "b2a975b1-786e-42dd-88fa-9bdd36a50ffe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:22.343765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.343737 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istiod-ca-cert" pod "b2a975b1-786e-42dd-88fa-9bdd36a50ffe" (UID: "b2a975b1-786e-42dd-88fa-9bdd36a50ffe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:22.345274 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.345248 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-kube-api-access-8v24s" pod "b2a975b1-786e-42dd-88fa-9bdd36a50ffe" (UID: "b2a975b1-786e-42dd-88fa-9bdd36a50ffe"). InnerVolumeSpecName "kube-api-access-8v24s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:22.345597 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.345565 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-envoy" pod "b2a975b1-786e-42dd-88fa-9bdd36a50ffe" (UID: "b2a975b1-786e-42dd-88fa-9bdd36a50ffe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:30:22.346423 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.346398 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-token" pod "b2a975b1-786e-42dd-88fa-9bdd36a50ffe" (UID: "b2a975b1-786e-42dd-88fa-9bdd36a50ffe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:22.358059 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.358021 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-podinfo" pod "b2a975b1-786e-42dd-88fa-9bdd36a50ffe" (UID: "b2a975b1-786e-42dd-88fa-9bdd36a50ffe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:30:22.443343 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.443302 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:22.443343 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.443345 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:22.443552 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.443360 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:22.443552 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.443377 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:22.443552 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.443391 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:22.443552 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.443405 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:22.443552 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.443419 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:22.443552 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.443435 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v24s\" (UniqueName: \"kubernetes.io/projected/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-kube-api-access-8v24s\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:22.443552 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.443450 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a975b1-786e-42dd-88fa-9bdd36a50ffe-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:22.486002 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.485966 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6"] Nov 02 22:30:22.493312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.493194 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-db36b079-29894310-lgsm6"] Nov 02 22:30:22.914215 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:22.914173 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:23.062555 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:23.062522 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" Nov 02 22:30:23.062731 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:23.062556 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb" event={"ID":"b2a975b1-786e-42dd-88fa-9bdd36a50ffe","Type":"ContainerDied","Data":"b90b5f8780998c97cb301768b865a1fdb06b8e5a31d8d73fded513de2aa3f5d2"} Nov 02 22:30:23.062731 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:23.062613 5871 scope.go:122] "RemoveContainer" containerID="420ae1da13c3ff5bc8f9c718225256aa39a3de25b215371f5de7b6ce07d7b09b" Nov 02 22:30:23.246426 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:23.246345 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb"] Nov 02 22:30:23.292633 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:23.292600 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7f1542e5-openshift-default-58c64cd459-57jtb"] Nov 02 22:30:23.314961 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:23.314932 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23fb37b7-5daf-4ef7-a553-3037abfe4a81" path="/var/lib/kubelet/pods/23fb37b7-5daf-4ef7-a553-3037abfe4a81/volumes" Nov 02 22:30:23.315516 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:23.315501 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a975b1-786e-42dd-88fa-9bdd36a50ffe" path="/var/lib/kubelet/pods/b2a975b1-786e-42dd-88fa-9bdd36a50ffe/volumes" Nov 02 22:30:23.643508 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:23.643476 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:24.014944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:24.014851 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:24.247100 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:24.247060 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:24.919562 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:24.919452 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:25.090142 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:25.090100 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:25.747930 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:25.741783 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:26.420896 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:26.420734 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:26.786077 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:26.785980 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jh4w5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:30:26.786362 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:26.786326 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jh4w5" podUID="440d8a6a-030e-4aad-a190-85fc0292617b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:30:27.110927 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:27.110606 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:27.740002 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:27.739965 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:29.689777 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:29.689738 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:31.534365 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:31.534323 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:31.751442 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:31.751409 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:32.224526 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:32.224488 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:32.413739 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:32.413705 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:34.952628 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:34.952595 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:39.399395 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:39.399355 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:39.979090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:39.979058 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:41.164496 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:41.164457 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:43.771614 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:30:43.771572 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:30:44.014158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.014113 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk"] Nov 02 22:30:44.014510 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.014449 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" podUID="7bd4c657-c3f1-424c-9d64-e821cdf2eb2a" containerName="maas-api" containerID="cri-o://bd2ea7052b7c75ea5725e66bb39a56b50d906ebea6e55777f513330468203e53" gracePeriod=30 Nov 02 22:30:44.148866 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.148812 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:44.317728 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.317693 5871 generic.go:356] "Generic (PLEG): container finished" podID="7bd4c657-c3f1-424c-9d64-e821cdf2eb2a" containerID="bd2ea7052b7c75ea5725e66bb39a56b50d906ebea6e55777f513330468203e53" exitCode=0 Nov 02 22:30:44.318210 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.317759 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" event={"ID":"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a","Type":"ContainerDied","Data":"bd2ea7052b7c75ea5725e66bb39a56b50d906ebea6e55777f513330468203e53"} Nov 02 22:30:44.351936 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.351908 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:30:44.445371 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.445330 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-kube-api-access-9w4pz\" (UniqueName: \"kubernetes.io/projected/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-kube-api-access-9w4pz\") pod \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " Nov 02 22:30:44.445590 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.445488 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-metrics-certs\") pod \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " Nov 02 22:30:44.445676 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.445633 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-maas-api-tls\") pod \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " Nov 02 22:30:44.445735 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.445719 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-openshift-service-ca\") pod \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\" (UID: \"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a\") " Nov 02 22:30:44.446732 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.446698 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-openshift-service-ca" pod "7bd4c657-c3f1-424c-9d64-e821cdf2eb2a" (UID: "7bd4c657-c3f1-424c-9d64-e821cdf2eb2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:44.464315 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.448945 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-kube-api-access-9w4pz" pod "7bd4c657-c3f1-424c-9d64-e821cdf2eb2a" (UID: "7bd4c657-c3f1-424c-9d64-e821cdf2eb2a"). InnerVolumeSpecName "kube-api-access-9w4pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:44.482628 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.481745 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-maas-api-tls" pod "7bd4c657-c3f1-424c-9d64-e821cdf2eb2a" (UID: "7bd4c657-c3f1-424c-9d64-e821cdf2eb2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:44.488069 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.488031 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-metrics-certs" pod "7bd4c657-c3f1-424c-9d64-e821cdf2eb2a" (UID: "7bd4c657-c3f1-424c-9d64-e821cdf2eb2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:44.547552 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.547519 5871 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-metrics-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:44.547552 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.547550 5871 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-maas-api-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:44.547769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.547564 5871 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-openshift-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:44.547769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.547575 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w4pz\" (UniqueName: \"kubernetes.io/projected/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a-kube-api-access-9w4pz\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:44.990895 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:44.989637 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:45.327279 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:45.327255 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" Nov 02 22:30:45.327439 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:45.327288 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk" event={"ID":"7bd4c657-c3f1-424c-9d64-e821cdf2eb2a","Type":"ContainerDied","Data":"7ade812063bd1365a9f2bfaf236a5c5ceee4114d6a6c8f28ec7039f6dc522e66"} Nov 02 22:30:45.327439 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:45.327349 5871 scope.go:122] "RemoveContainer" containerID="bd2ea7052b7c75ea5725e66bb39a56b50d906ebea6e55777f513330468203e53" Nov 02 22:30:45.494651 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:45.494610 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk"] Nov 02 22:30:45.544775 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:45.544738 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e86dfb1-64bd579cc6-wlxzk"] Nov 02 22:30:45.790231 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:45.790192 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:46.060400 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:46.060295 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:46.245341 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:46.245308 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:46.421075 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:46.421037 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:47.315496 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:47.315463 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd4c657-c3f1-424c-9d64-e821cdf2eb2a" path="/var/lib/kubelet/pods/7bd4c657-c3f1-424c-9d64-e821cdf2eb2a/volumes" Nov 02 22:30:47.350522 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:47.350491 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:47.525965 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:47.525929 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:51.410020 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:51.409983 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:30:55.288439 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:55.288391 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:30:56.477821 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:30:56.477782 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:00.988363 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.988323 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b"] Nov 02 22:31:00.989021 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989005 5871 cpu_manager.go:414] "containerMap: removing container" podUID="b2a975b1-786e-42dd-88fa-9bdd36a50ffe" containerName="istio-proxy" Nov 02 22:31:00.989069 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989022 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a975b1-786e-42dd-88fa-9bdd36a50ffe" containerName="istio-proxy" Nov 02 22:31:00.989069 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989036 5871 cpu_manager.go:414] "containerMap: removing container" podUID="2e9d9f83-67f8-44dd-89a6-1c622fd4708a" containerName="istio-proxy" Nov 02 22:31:00.989069 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989044 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e9d9f83-67f8-44dd-89a6-1c622fd4708a" containerName="istio-proxy" Nov 02 22:31:00.989069 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989058 5871 cpu_manager.go:414] "containerMap: removing container" podUID="23fb37b7-5daf-4ef7-a553-3037abfe4a81" containerName="cleanup" Nov 02 22:31:00.989069 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989063 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23fb37b7-5daf-4ef7-a553-3037abfe4a81" containerName="cleanup" Nov 02 22:31:00.989069 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989070 5871 cpu_manager.go:414] "containerMap: removing container" podUID="23fb37b7-5daf-4ef7-a553-3037abfe4a81" containerName="cleanup" Nov 02 22:31:00.989269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989075 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23fb37b7-5daf-4ef7-a553-3037abfe4a81" containerName="cleanup" Nov 02 22:31:00.989269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989085 5871 cpu_manager.go:414] "containerMap: removing container" podUID="c910d204-821e-4a39-9f51-40668aaa2e7e" containerName="istio-proxy" Nov 02 22:31:00.989269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989090 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c910d204-821e-4a39-9f51-40668aaa2e7e" containerName="istio-proxy" Nov 02 22:31:00.989269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989107 5871 cpu_manager.go:414] "containerMap: removing container" podUID="9b8e2845-337e-4fa5-b745-ae219273156b" containerName="istio-proxy" Nov 02 22:31:00.989269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989114 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8e2845-337e-4fa5-b745-ae219273156b" containerName="istio-proxy" Nov 02 22:31:00.989269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989125 5871 cpu_manager.go:414] "containerMap: removing container" podUID="7bd4c657-c3f1-424c-9d64-e821cdf2eb2a" containerName="maas-api" Nov 02 22:31:00.989269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989129 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd4c657-c3f1-424c-9d64-e821cdf2eb2a" containerName="maas-api" Nov 02 22:31:00.989269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989260 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a975b1-786e-42dd-88fa-9bdd36a50ffe" containerName="istio-proxy" Nov 02 22:31:00.989269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989267 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a975b1-786e-42dd-88fa-9bdd36a50ffe" containerName="istio-proxy" Nov 02 22:31:00.989541 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989275 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e9d9f83-67f8-44dd-89a6-1c622fd4708a" containerName="istio-proxy" Nov 02 22:31:00.989541 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989283 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="23fb37b7-5daf-4ef7-a553-3037abfe4a81" containerName="cleanup" Nov 02 22:31:00.989541 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989289 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="23fb37b7-5daf-4ef7-a553-3037abfe4a81" containerName="cleanup" Nov 02 22:31:00.989541 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989295 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="c910d204-821e-4a39-9f51-40668aaa2e7e" containerName="istio-proxy" Nov 02 22:31:00.989541 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989304 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8e2845-337e-4fa5-b745-ae219273156b" containerName="istio-proxy" Nov 02 22:31:00.989541 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.989310 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd4c657-c3f1-424c-9d64-e821cdf2eb2a" containerName="maas-api" Nov 02 22:31:00.994804 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:00.994782 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:01.048318 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.048281 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b"] Nov 02 22:31:01.085311 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.085264 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-metrics-certs\") pod \"maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:01.085523 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.085393 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26990ef1-f003-4196-b347-9dbf761a25ec-openshift-service-ca\") pod \"maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:01.085598 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.085559 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-maas-api-tls\") pod \"maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:01.085598 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.085590 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9psh\" (UniqueName: \"kubernetes.io/projected/26990ef1-f003-4196-b347-9dbf761a25ec-kube-api-access-c9psh\") pod \"maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:01.093539 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.093511 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-2663d62e-serving-cert\"" Nov 02 22:31:01.187161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.187125 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26990ef1-f003-4196-b347-9dbf761a25ec-openshift-service-ca\") pod \"maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:01.187357 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.187249 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-maas-api-tls\") pod \"maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:01.187357 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.187269 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9psh\" (UniqueName: \"kubernetes.io/projected/26990ef1-f003-4196-b347-9dbf761a25ec-kube-api-access-c9psh\") pod \"maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:01.187494 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:31:01.187397 5871 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-serving-cert: secret "maas-api-e2e-migrate-2663d62e-serving-cert" not found Nov 02 22:31:01.187494 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:31:01.187481 5871 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-maas-api-tls podName:26990ef1-f003-4196-b347-9dbf761a25ec nodeName:}" failed. No retries permitted until 2026-11-02 22:31:01.687458013 +0000 UTC m=+1484.499788185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-maas-api-tls") pod "maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" (UID: "26990ef1-f003-4196-b347-9dbf761a25ec") : secret "maas-api-e2e-migrate-2663d62e-serving-cert" not found Nov 02 22:31:01.187614 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.187502 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-metrics-certs\") pod \"maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:01.187946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.187907 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26990ef1-f003-4196-b347-9dbf761a25ec-openshift-service-ca\") pod \"maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:01.189969 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.189945 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-metrics-certs\") pod \"maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:01.217477 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.217435 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:01.250282 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.250164 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9psh\" (UniqueName: \"kubernetes.io/projected/26990ef1-f003-4196-b347-9dbf761a25ec-kube-api-access-c9psh\") pod \"maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:01.394220 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.394183 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:01.572524 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.572396 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:01.695576 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.695542 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-maas-api-tls\") pod \"maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:01.698111 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.698081 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-maas-api-tls\") pod \"maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:01.908962 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:01.908925 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:02.294688 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:02.294654 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b"] Nov 02 22:31:02.296961 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:31:02.296935 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26990ef1_f003_4196_b347_9dbf761a25ec.slice/crio-a18a16a3b4743be77815ae29a3b9a57e55cdf5d7abe266a06c0e7235b1d941d9 WatchSource:0}: Error finding container a18a16a3b4743be77815ae29a3b9a57e55cdf5d7abe266a06c0e7235b1d941d9: Status 404 returned error can't find the container with id a18a16a3b4743be77815ae29a3b9a57e55cdf5d7abe266a06c0e7235b1d941d9 Nov 02 22:31:02.470938 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:02.470899 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" event={"ID":"26990ef1-f003-4196-b347-9dbf761a25ec","Type":"ContainerStarted","Data":"a18a16a3b4743be77815ae29a3b9a57e55cdf5d7abe266a06c0e7235b1d941d9"} Nov 02 22:31:03.481808 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:03.481706 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" event={"ID":"26990ef1-f003-4196-b347-9dbf761a25ec","Type":"ContainerStarted","Data":"f0fd8931639cdd84e03479096472e1aef215c828eac645139b42a6f638fcde93"} Nov 02 22:31:03.481808 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:03.481792 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:03.607277 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:03.607201 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" podStartSLOduration=2.694174115 podStartE2EDuration="3.607179102s" totalImagesPullingTime="913.004987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:31:03.601630641 +0000 UTC m=+1486.413960813" watchObservedRunningTime="2026-11-02 22:31:03.607179102 +0000 UTC m=+1486.419509280" Nov 02 22:31:05.498465 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:05.498432 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:05.995684 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:05.995646 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:06.170141 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:06.170104 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:06.575303 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:06.575264 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:06.759555 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:06.759514 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:09.496274 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:09.496240 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:31:10.942138 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:10.942106 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:11.622650 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:11.622616 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:13.770770 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:31:13.770731 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:31:15.409994 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.409957 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp"] Nov 02 22:31:15.410668 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.410650 5871 cpu_manager.go:414] "containerMap: removing container" podUID="b2a975b1-786e-42dd-88fa-9bdd36a50ffe" containerName="istio-proxy" Nov 02 22:31:15.410707 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.410669 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a975b1-786e-42dd-88fa-9bdd36a50ffe" containerName="istio-proxy" Nov 02 22:31:15.417019 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.416985 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:15.473465 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.473431 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-fb670683-serving-cert\"" Nov 02 22:31:15.524718 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.524672 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-openshift-service-ca\") pod \"maas-api-e2e-ait-fb670683-6dc5b77-p2qxp\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:15.524963 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.524829 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg825\" (UniqueName: \"kubernetes.io/projected/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-kube-api-access-lg825\") pod \"maas-api-e2e-ait-fb670683-6dc5b77-p2qxp\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:15.524963 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.524942 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-metrics-certs\") pod \"maas-api-e2e-ait-fb670683-6dc5b77-p2qxp\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:15.525090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.525076 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-maas-api-tls\") pod \"maas-api-e2e-ait-fb670683-6dc5b77-p2qxp\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:15.527152 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.527127 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp"] Nov 02 22:31:15.627204 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.627167 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-maas-api-tls\") pod \"maas-api-e2e-ait-fb670683-6dc5b77-p2qxp\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:15.627416 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.627267 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-openshift-service-ca\") pod \"maas-api-e2e-ait-fb670683-6dc5b77-p2qxp\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:15.627416 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:31:15.627334 5871 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-serving-cert: secret "maas-api-e2e-ait-fb670683-serving-cert" not found Nov 02 22:31:15.627535 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.627417 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg825\" (UniqueName: \"kubernetes.io/projected/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-kube-api-access-lg825\") pod \"maas-api-e2e-ait-fb670683-6dc5b77-p2qxp\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:15.627535 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:31:15.627445 5871 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-maas-api-tls podName:20f6517a-1f4a-4eb9-b7cc-4b02d76cacae nodeName:}" failed. No retries permitted until 2026-11-02 22:31:16.127424397 +0000 UTC m=+1498.939754775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-maas-api-tls") pod "maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" (UID: "20f6517a-1f4a-4eb9-b7cc-4b02d76cacae") : secret "maas-api-e2e-ait-fb670683-serving-cert" not found Nov 02 22:31:15.627657 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.627541 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-metrics-certs\") pod \"maas-api-e2e-ait-fb670683-6dc5b77-p2qxp\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:15.628236 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.628204 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-openshift-service-ca\") pod \"maas-api-e2e-ait-fb670683-6dc5b77-p2qxp\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:15.630765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.630739 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-metrics-certs\") pod \"maas-api-e2e-ait-fb670683-6dc5b77-p2qxp\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:15.640016 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.639979 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:15.694362 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.693920 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg825\" (UniqueName: \"kubernetes.io/projected/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-kube-api-access-lg825\") pod \"maas-api-e2e-ait-fb670683-6dc5b77-p2qxp\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:15.822967 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:15.822926 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:16.006485 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:16.006397 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:16.134859 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:16.134813 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-maas-api-tls\") pod \"maas-api-e2e-ait-fb670683-6dc5b77-p2qxp\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:16.137298 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:16.137270 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-maas-api-tls\") pod \"maas-api-e2e-ait-fb670683-6dc5b77-p2qxp\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:16.332944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:16.332851 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:16.718233 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:16.718205 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp"] Nov 02 22:31:16.719846 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:31:16.719813 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f6517a_1f4a_4eb9_b7cc_4b02d76cacae.slice/crio-0ede54770d2f36afb59f81fb3736c6d0ec16c1d98f1b0755f32e1690ab17fd26 WatchSource:0}: Error finding container 0ede54770d2f36afb59f81fb3736c6d0ec16c1d98f1b0755f32e1690ab17fd26: Status 404 returned error can't find the container with id 0ede54770d2f36afb59f81fb3736c6d0ec16c1d98f1b0755f32e1690ab17fd26 Nov 02 22:31:17.602270 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:17.602223 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" event={"ID":"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae","Type":"ContainerStarted","Data":"0ede54770d2f36afb59f81fb3736c6d0ec16c1d98f1b0755f32e1690ab17fd26"} Nov 02 22:31:18.611527 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:18.611485 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" event={"ID":"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae","Type":"ContainerStarted","Data":"11488850587f339b5e46d00fe743c44f64a6068c88e13e52dda04bbd370fcaac"} Nov 02 22:31:18.612209 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:18.611905 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:18.732184 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:18.732126 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" podStartSLOduration=2.665412569 podStartE2EDuration="3.732109672s" totalImagesPullingTime="1.066697103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:31:18.730156589 +0000 UTC m=+1501.542486771" watchObservedRunningTime="2026-11-02 22:31:18.732109672 +0000 UTC m=+1501.544439850" Nov 02 22:31:20.335010 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.334975 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8"] Nov 02 22:31:20.339995 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.339971 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:31:20.393890 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.393839 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8"] Nov 02 22:31:20.439474 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.439437 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bb1a7ce7-serving-cert\"" Nov 02 22:31:20.500842 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.500804 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7a6cfc-826c-4c21-8d77-35272c34d117-openshift-service-ca\") pod \"maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8\" (UID: \"7c7a6cfc-826c-4c21-8d77-35272c34d117\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:31:20.501058 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.500917 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7a6cfc-826c-4c21-8d77-35272c34d117-maas-api-tls\") pod \"maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8\" (UID: \"7c7a6cfc-826c-4c21-8d77-35272c34d117\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:31:20.501192 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.501154 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7a6cfc-826c-4c21-8d77-35272c34d117-metrics-certs\") pod \"maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8\" (UID: \"7c7a6cfc-826c-4c21-8d77-35272c34d117\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:31:20.501400 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.501383 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h5z2\" (UniqueName: \"kubernetes.io/projected/7c7a6cfc-826c-4c21-8d77-35272c34d117-kube-api-access-7h5z2\") pod \"maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8\" (UID: \"7c7a6cfc-826c-4c21-8d77-35272c34d117\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:31:20.567970 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.567908 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:20.603138 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.603039 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h5z2\" (UniqueName: \"kubernetes.io/projected/7c7a6cfc-826c-4c21-8d77-35272c34d117-kube-api-access-7h5z2\") pod \"maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8\" (UID: \"7c7a6cfc-826c-4c21-8d77-35272c34d117\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:31:20.603315 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.603144 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7a6cfc-826c-4c21-8d77-35272c34d117-openshift-service-ca\") pod \"maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8\" (UID: \"7c7a6cfc-826c-4c21-8d77-35272c34d117\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:31:20.603315 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.603205 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7a6cfc-826c-4c21-8d77-35272c34d117-maas-api-tls\") pod \"maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8\" (UID: \"7c7a6cfc-826c-4c21-8d77-35272c34d117\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:31:20.603315 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.603271 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7a6cfc-826c-4c21-8d77-35272c34d117-metrics-certs\") pod \"maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8\" (UID: \"7c7a6cfc-826c-4c21-8d77-35272c34d117\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:31:20.604162 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.604126 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7a6cfc-826c-4c21-8d77-35272c34d117-openshift-service-ca\") pod \"maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8\" (UID: \"7c7a6cfc-826c-4c21-8d77-35272c34d117\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:31:20.605844 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.605822 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7a6cfc-826c-4c21-8d77-35272c34d117-metrics-certs\") pod \"maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8\" (UID: \"7c7a6cfc-826c-4c21-8d77-35272c34d117\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:31:20.605995 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.605887 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7a6cfc-826c-4c21-8d77-35272c34d117-maas-api-tls\") pod \"maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8\" (UID: \"7c7a6cfc-826c-4c21-8d77-35272c34d117\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:31:20.665826 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.665782 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h5z2\" (UniqueName: \"kubernetes.io/projected/7c7a6cfc-826c-4c21-8d77-35272c34d117-kube-api-access-7h5z2\") pod \"maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8\" (UID: \"7c7a6cfc-826c-4c21-8d77-35272c34d117\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:31:20.745569 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.745533 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:20.921304 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.921270 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:20.954926 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:20.954883 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:31:21.345261 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:21.345223 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8"] Nov 02 22:31:21.347047 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:31:21.347016 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7a6cfc_826c_4c21_8d77_35272c34d117.slice/crio-3c4aa5d42797657e1edf78eb3db14aa8035f660cd0ca9934b763717793f145de WatchSource:0}: Error finding container 3c4aa5d42797657e1edf78eb3db14aa8035f660cd0ca9934b763717793f145de: Status 404 returned error can't find the container with id 3c4aa5d42797657e1edf78eb3db14aa8035f660cd0ca9934b763717793f145de Nov 02 22:31:21.713340 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:21.713278 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:21.759694 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:21.759635 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" event={"ID":"7c7a6cfc-826c-4c21-8d77-35272c34d117","Type":"ContainerStarted","Data":"3c4aa5d42797657e1edf78eb3db14aa8035f660cd0ca9934b763717793f145de"} Nov 02 22:31:21.862994 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:21.862954 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:31:21.894939 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:21.888003 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:22.774587 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:22.774540 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" event={"ID":"7c7a6cfc-826c-4c21-8d77-35272c34d117","Type":"ContainerStarted","Data":"67d4f5649c2aa65fd9fc77e0508b41472f00baf1eaad337469414e6640a78683"} Nov 02 22:31:22.775144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:22.774641 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:31:22.896281 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:22.896217 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" podStartSLOduration=1.952579277 podStartE2EDuration="2.896199431s" totalImagesPullingTime="943.620154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:31:22.891936119 +0000 UTC m=+1505.704266298" watchObservedRunningTime="2026-11-02 22:31:22.896199431 +0000 UTC m=+1505.708529609" Nov 02 22:31:24.628849 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:24.628817 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:31:26.082255 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:26.082212 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:26.524062 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:26.523995 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:26.736616 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:26.736580 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:28.788785 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:28.788753 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:31:31.272792 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:31.272737 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:36.086589 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:36.086534 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:38.342245 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:38.342206 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:38.524072 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:38.524042 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:39.981269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:39.981223 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:40.182061 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:40.182022 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:40.406512 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:40.406424 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:41.656503 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:41.656461 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:41.834023 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:41.833981 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:43.772134 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:31:43.772093 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:31:45.427268 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:45.427224 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:46.047914 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:46.047863 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:46.581893 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:46.581843 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:46.762844 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:46.762806 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:46.939353 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:46.939300 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:51.119086 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:51.119035 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:51.783494 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:51.783451 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff"] Nov 02 22:31:51.806339 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:51.806303 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv"] Nov 02 22:31:51.872434 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:51.872398 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:52.060465 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:52.060372 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:56.679087 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:56.678750 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:58.548129 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:58.548080 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:58.722441 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:58.722408 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:31:59.985282 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:31:59.985234 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:32:02.077815 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:02.077778 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:02.262445 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:02.262411 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:02.444543 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:02.444506 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:06.522616 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:06.522578 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:06.807092 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:06.806995 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" containerID="cri-o://dd266b49e8354c4975453b18d78102174471c973a05c7cf985ab56a07348b01e" gracePeriod=45 Nov 02 22:32:06.827768 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:06.827730 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" containerID="cri-o://c54b7568f10299d80d9027850cf09d654ae157a0af94ee39c434140c53ca32a8" gracePeriod=45 Nov 02 22:32:07.256146 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.256084 5871 generic.go:356] "Generic (PLEG): container finished" podID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerID="c54b7568f10299d80d9027850cf09d654ae157a0af94ee39c434140c53ca32a8" exitCode=0 Nov 02 22:32:07.256660 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.256630 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" event={"ID":"180c8830-c2d7-42b4-9a5e-41f15996a8d6","Type":"ContainerDied","Data":"c54b7568f10299d80d9027850cf09d654ae157a0af94ee39c434140c53ca32a8"} Nov 02 22:32:07.263195 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.263157 5871 generic.go:356] "Generic (PLEG): container finished" podID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerID="dd266b49e8354c4975453b18d78102174471c973a05c7cf985ab56a07348b01e" exitCode=0 Nov 02 22:32:07.263424 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.263400 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" event={"ID":"b4c1aa58-b96e-4539-b445-62d8cb67ffa9","Type":"ContainerDied","Data":"dd266b49e8354c4975453b18d78102174471c973a05c7cf985ab56a07348b01e"} Nov 02 22:32:07.291931 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.291081 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:32:07.293472 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.293447 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:32:07.361156 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.361120 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-kube-api-access-n2n7j\" (UniqueName: \"kubernetes.io/projected/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-kube-api-access-n2n7j\") pod \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " Nov 02 22:32:07.363886 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.363842 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-kube-api-access-n2n7j" pod "b4c1aa58-b96e-4539-b445-62d8cb67ffa9" (UID: "b4c1aa58-b96e-4539-b445-62d8cb67ffa9"). InnerVolumeSpecName "kube-api-access-n2n7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:07.462568 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.462535 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-dshm\") pod \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " Nov 02 22:32:07.462779 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.462587 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/180c8830-c2d7-42b4-9a5e-41f15996a8d6-kube-api-access-mszns\" (UniqueName: \"kubernetes.io/projected/180c8830-c2d7-42b4-9a5e-41f15996a8d6-kube-api-access-mszns\") pod \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " Nov 02 22:32:07.462779 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.462622 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-tmp-dir\") pod \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " Nov 02 22:32:07.462779 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.462690 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-model-cache\") pod \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " Nov 02 22:32:07.462779 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.462767 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-tls-certs\") pod \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " Nov 02 22:32:07.463037 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.462808 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/180c8830-c2d7-42b4-9a5e-41f15996a8d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/180c8830-c2d7-42b4-9a5e-41f15996a8d6-tls-certs\") pod \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " Nov 02 22:32:07.463037 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.462842 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-home\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-home\") pod \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " Nov 02 22:32:07.463037 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.462888 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-home\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-home\") pod \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " Nov 02 22:32:07.463037 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.462942 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-dshm\") pod \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\" (UID: \"b4c1aa58-b96e-4539-b445-62d8cb67ffa9\") " Nov 02 22:32:07.463037 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.462978 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-tmp-dir\") pod \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " Nov 02 22:32:07.463305 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.463078 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-model-cache\") pod \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\" (UID: \"180c8830-c2d7-42b4-9a5e-41f15996a8d6\") " Nov 02 22:32:07.463305 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.463121 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-model-cache" pod "b4c1aa58-b96e-4539-b445-62d8cb67ffa9" (UID: "b4c1aa58-b96e-4539-b445-62d8cb67ffa9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:07.463573 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.463547 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-tmp-dir" pod "b4c1aa58-b96e-4539-b445-62d8cb67ffa9" (UID: "b4c1aa58-b96e-4539-b445-62d8cb67ffa9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:07.463759 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.463738 5871 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-model-cache\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.463840 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.463762 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-model-cache" pod "180c8830-c2d7-42b4-9a5e-41f15996a8d6" (UID: "180c8830-c2d7-42b4-9a5e-41f15996a8d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:07.463840 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.463766 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2n7j\" (UniqueName: \"kubernetes.io/projected/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-kube-api-access-n2n7j\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.463840 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.463806 5871 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-tmp-dir\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.464116 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.464028 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-home" pod "180c8830-c2d7-42b4-9a5e-41f15996a8d6" (UID: "180c8830-c2d7-42b4-9a5e-41f15996a8d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:07.464116 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.464041 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-home" pod "b4c1aa58-b96e-4539-b445-62d8cb67ffa9" (UID: "b4c1aa58-b96e-4539-b445-62d8cb67ffa9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:07.466186 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.466158 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-tmp-dir" pod "180c8830-c2d7-42b4-9a5e-41f15996a8d6" (UID: "180c8830-c2d7-42b4-9a5e-41f15996a8d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:07.467047 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.467015 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-dshm" pod "b4c1aa58-b96e-4539-b445-62d8cb67ffa9" (UID: "b4c1aa58-b96e-4539-b445-62d8cb67ffa9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:07.467202 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.467169 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-tls-certs" pod "b4c1aa58-b96e-4539-b445-62d8cb67ffa9" (UID: "b4c1aa58-b96e-4539-b445-62d8cb67ffa9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:07.469115 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.469045 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-dshm" pod "180c8830-c2d7-42b4-9a5e-41f15996a8d6" (UID: "180c8830-c2d7-42b4-9a5e-41f15996a8d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:07.469115 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.469101 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180c8830-c2d7-42b4-9a5e-41f15996a8d6-kube-api-access-mszns" pod "180c8830-c2d7-42b4-9a5e-41f15996a8d6" (UID: "180c8830-c2d7-42b4-9a5e-41f15996a8d6"). InnerVolumeSpecName "kube-api-access-mszns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:07.470332 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.470310 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/180c8830-c2d7-42b4-9a5e-41f15996a8d6-tls-certs" pod "180c8830-c2d7-42b4-9a5e-41f15996a8d6" (UID: "180c8830-c2d7-42b4-9a5e-41f15996a8d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:07.565645 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.565606 5871 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-home\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.565645 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.565643 5871 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-home\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.566128 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.565655 5871 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-dshm\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.566128 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.565669 5871 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-tmp-dir\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.566128 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.565680 5871 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-model-cache\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.566128 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.565691 5871 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/180c8830-c2d7-42b4-9a5e-41f15996a8d6-dshm\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.566128 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.565707 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mszns\" (UniqueName: \"kubernetes.io/projected/180c8830-c2d7-42b4-9a5e-41f15996a8d6-kube-api-access-mszns\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.566128 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.565720 5871 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4c1aa58-b96e-4539-b445-62d8cb67ffa9-tls-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:07.566128 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:07.565735 5871 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/180c8830-c2d7-42b4-9a5e-41f15996a8d6-tls-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:08.117697 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:08.117228 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:08.273799 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:08.273769 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" Nov 02 22:32:08.274029 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:08.273805 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" event={"ID":"180c8830-c2d7-42b4-9a5e-41f15996a8d6","Type":"ContainerDied","Data":"4c2f3d3c77db39d9dd754e3adfbb65248e9ec83e3cb1c8c957be5beadf8e7f13"} Nov 02 22:32:08.274029 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:08.273868 5871 scope.go:122] "RemoveContainer" containerID="c54b7568f10299d80d9027850cf09d654ae157a0af94ee39c434140c53ca32a8" Nov 02 22:32:08.275931 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:08.275906 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" Nov 02 22:32:08.276064 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:08.275909 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff" event={"ID":"b4c1aa58-b96e-4539-b445-62d8cb67ffa9","Type":"ContainerDied","Data":"c8c6b5f97ef1c4317e2fdc56e21b749a79124fc3480cce1aa7d78966092f2389"} Nov 02 22:32:08.283257 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:08.283217 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:32:08.290862 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:08.290647 5871 scope.go:122] "RemoveContainer" containerID="dd266b49e8354c4975453b18d78102174471c973a05c7cf985ab56a07348b01e" Nov 02 22:32:08.441568 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:08.441477 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff"] Nov 02 22:32:08.448349 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:08.448314 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/distinct-2-w1-ca85db-kserve-8bcc54f7d-rbsff"] Nov 02 22:32:08.559078 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:08.559039 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv"] Nov 02 22:32:08.563312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:08.563283 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-ca85db/simulator-w1-ca85db-kserve-77795c999-mnstv"] Nov 02 22:32:09.314683 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:09.314647 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" path="/var/lib/kubelet/pods/180c8830-c2d7-42b4-9a5e-41f15996a8d6/volumes" Nov 02 22:32:09.315199 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:09.315184 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" path="/var/lib/kubelet/pods/b4c1aa58-b96e-4539-b445-62d8cb67ffa9/volumes" Nov 02 22:32:12.303075 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:12.303037 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:12.475993 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:12.475952 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:13.778831 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:32:13.778770 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:32:13.927610 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:13.926769 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:19.292583 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:19.292544 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:19.472391 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:19.472353 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:20.239937 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:20.239904 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:20.419181 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:20.419143 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:20.931387 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:20.931351 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:22.680332 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:22.680289 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:22.859437 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:22.859396 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:24.327376 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:24.327339 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:27.059324 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:27.059292 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:28.113526 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:28.113472 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:28.341803 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:28.341761 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:29.277283 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:29.277245 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6"] Nov 02 22:32:29.277727 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:29.277474 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" podUID="5c65b13b-f4dd-4a8a-9844-a0f0309be485" containerName="istio-proxy" containerID="cri-o://848f78054297f0d31c65fc30cdd75611291fe6264583b21b81ba90a9b5f7baba" gracePeriod=30 Nov 02 22:32:29.288415 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:29.288380 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:30.132424 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:30.132390 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:30.403767 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:30.403682 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:30.581757 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:30.581721 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:30.760357 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:30.760268 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:34.177238 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.177186 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:34.520079 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.520047 5871 generic.go:356] "Generic (PLEG): container finished" podID="5c65b13b-f4dd-4a8a-9844-a0f0309be485" containerID="848f78054297f0d31c65fc30cdd75611291fe6264583b21b81ba90a9b5f7baba" exitCode=0 Nov 02 22:32:34.520281 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.520269 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" event={"ID":"5c65b13b-f4dd-4a8a-9844-a0f0309be485","Type":"ContainerDied","Data":"848f78054297f0d31c65fc30cdd75611291fe6264583b21b81ba90a9b5f7baba"} Nov 02 22:32:34.562805 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.562775 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:32:34.725121 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.725016 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istiod-ca-cert\") pod \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " Nov 02 22:32:34.725290 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.725136 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-data\") pod \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " Nov 02 22:32:34.725290 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.725189 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c65b13b-f4dd-4a8a-9844-a0f0309be485-kube-api-access-kz9fh\" (UniqueName: \"kubernetes.io/projected/5c65b13b-f4dd-4a8a-9844-a0f0309be485-kube-api-access-kz9fh\") pod \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " Nov 02 22:32:34.725744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.725397 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-podinfo\") pod \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " Nov 02 22:32:34.725744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.725464 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istiod-ca-cert" pod "5c65b13b-f4dd-4a8a-9844-a0f0309be485" (UID: "5c65b13b-f4dd-4a8a-9844-a0f0309be485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:34.725744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.725502 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-data" pod "5c65b13b-f4dd-4a8a-9844-a0f0309be485" (UID: "5c65b13b-f4dd-4a8a-9844-a0f0309be485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:34.725744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.725631 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-envoy\") pod \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " Nov 02 22:32:34.726439 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.726416 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-workload-certs\") pod \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " Nov 02 22:32:34.726669 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.726647 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-workload-certs" pod "5c65b13b-f4dd-4a8a-9844-a0f0309be485" (UID: "5c65b13b-f4dd-4a8a-9844-a0f0309be485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:34.726884 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.726837 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-workload-socket\") pod \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " Nov 02 22:32:34.727134 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.726940 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-workload-socket" pod "5c65b13b-f4dd-4a8a-9844-a0f0309be485" (UID: "5c65b13b-f4dd-4a8a-9844-a0f0309be485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:34.727774 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.727322 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-token\" (UniqueName: \"kubernetes.io/projected/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-token\") pod \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " Nov 02 22:32:34.728522 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.728484 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-credential-socket" pod "5c65b13b-f4dd-4a8a-9844-a0f0309be485" (UID: "5c65b13b-f4dd-4a8a-9844-a0f0309be485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:34.728679 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.728662 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-credential-socket\") pod \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\" (UID: \"5c65b13b-f4dd-4a8a-9844-a0f0309be485\") " Nov 02 22:32:34.729006 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.728975 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-envoy" pod "5c65b13b-f4dd-4a8a-9844-a0f0309be485" (UID: "5c65b13b-f4dd-4a8a-9844-a0f0309be485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:34.729766 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.729745 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:34.729923 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.729771 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:34.729923 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.729787 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:34.729923 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.729802 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:34.729923 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.729820 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:34.729923 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.729833 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:34.730472 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.730200 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c65b13b-f4dd-4a8a-9844-a0f0309be485-kube-api-access-kz9fh" pod "5c65b13b-f4dd-4a8a-9844-a0f0309be485" (UID: "5c65b13b-f4dd-4a8a-9844-a0f0309be485"). InnerVolumeSpecName "kube-api-access-kz9fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:34.733779 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.733754 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-token" pod "5c65b13b-f4dd-4a8a-9844-a0f0309be485" (UID: "5c65b13b-f4dd-4a8a-9844-a0f0309be485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:34.744640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.744586 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-podinfo" pod "5c65b13b-f4dd-4a8a-9844-a0f0309be485" (UID: "5c65b13b-f4dd-4a8a-9844-a0f0309be485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:32:34.830839 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.830795 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz9fh\" (UniqueName: \"kubernetes.io/projected/5c65b13b-f4dd-4a8a-9844-a0f0309be485-kube-api-access-kz9fh\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:34.830839 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.830834 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:34.830839 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:34.830844 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c65b13b-f4dd-4a8a-9844-a0f0309be485-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:35.528551 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:35.528519 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" Nov 02 22:32:35.528551 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:35.528531 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6" event={"ID":"5c65b13b-f4dd-4a8a-9844-a0f0309be485","Type":"ContainerDied","Data":"83645f9910503cfdf723d16b4db8395ed831fb72002892e1f2f55a69684e332c"} Nov 02 22:32:35.529050 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:35.528596 5871 scope.go:122] "RemoveContainer" containerID="848f78054297f0d31c65fc30cdd75611291fe6264583b21b81ba90a9b5f7baba" Nov 02 22:32:35.706647 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:35.706574 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:35.750361 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:35.750323 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6"] Nov 02 22:32:35.799845 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:35.799803 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-65rr6"] Nov 02 22:32:36.191934 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:36.191904 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:36.387106 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:36.387067 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:37.221700 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:37.221663 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:37.315501 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:37.315456 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c65b13b-f4dd-4a8a-9844-a0f0309be485" path="/var/lib/kubelet/pods/5c65b13b-f4dd-4a8a-9844-a0f0309be485/volumes" Nov 02 22:32:37.472480 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:37.472387 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:37.688943 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:37.688895 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:32:38.830752 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:38.830712 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b"] Nov 02 22:32:38.831455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:38.831045 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" podUID="26990ef1-f003-4196-b347-9dbf761a25ec" containerName="maas-api" containerID="cri-o://f0fd8931639cdd84e03479096472e1aef215c828eac645139b42a6f638fcde93" gracePeriod=30 Nov 02 22:32:38.926285 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:38.926246 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:39.174345 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.174312 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:32:39.228724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.212729 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26990ef1-f003-4196-b347-9dbf761a25ec-kube-api-access-c9psh\" (UniqueName: \"kubernetes.io/projected/26990ef1-f003-4196-b347-9dbf761a25ec-kube-api-access-c9psh\") pod \"26990ef1-f003-4196-b347-9dbf761a25ec\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " Nov 02 22:32:39.228724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.213056 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26990ef1-f003-4196-b347-9dbf761a25ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26990ef1-f003-4196-b347-9dbf761a25ec-openshift-service-ca\") pod \"26990ef1-f003-4196-b347-9dbf761a25ec\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " Nov 02 22:32:39.228724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.213188 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-metrics-certs\") pod \"26990ef1-f003-4196-b347-9dbf761a25ec\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " Nov 02 22:32:39.228724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.213424 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-maas-api-tls\") pod \"26990ef1-f003-4196-b347-9dbf761a25ec\" (UID: \"26990ef1-f003-4196-b347-9dbf761a25ec\") " Nov 02 22:32:39.228724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.213443 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26990ef1-f003-4196-b347-9dbf761a25ec-openshift-service-ca" pod "26990ef1-f003-4196-b347-9dbf761a25ec" (UID: "26990ef1-f003-4196-b347-9dbf761a25ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:39.228724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.214562 5871 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26990ef1-f003-4196-b347-9dbf761a25ec-openshift-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:39.228724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.216049 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-maas-api-tls" pod "26990ef1-f003-4196-b347-9dbf761a25ec" (UID: "26990ef1-f003-4196-b347-9dbf761a25ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:39.228724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.216704 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-metrics-certs" pod "26990ef1-f003-4196-b347-9dbf761a25ec" (UID: "26990ef1-f003-4196-b347-9dbf761a25ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:39.228724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.216720 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26990ef1-f003-4196-b347-9dbf761a25ec-kube-api-access-c9psh" pod "26990ef1-f003-4196-b347-9dbf761a25ec" (UID: "26990ef1-f003-4196-b347-9dbf761a25ec"). InnerVolumeSpecName "kube-api-access-c9psh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:39.327359 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.327311 5871 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-metrics-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:39.327617 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.327595 5871 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26990ef1-f003-4196-b347-9dbf761a25ec-maas-api-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:39.327725 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.327713 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9psh\" (UniqueName: \"kubernetes.io/projected/26990ef1-f003-4196-b347-9dbf761a25ec-kube-api-access-c9psh\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:39.572173 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.572076 5871 generic.go:356] "Generic (PLEG): container finished" podID="26990ef1-f003-4196-b347-9dbf761a25ec" containerID="f0fd8931639cdd84e03479096472e1aef215c828eac645139b42a6f638fcde93" exitCode=0 Nov 02 22:32:39.572173 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.572128 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" Nov 02 22:32:39.572173 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.572158 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" event={"ID":"26990ef1-f003-4196-b347-9dbf761a25ec","Type":"ContainerDied","Data":"f0fd8931639cdd84e03479096472e1aef215c828eac645139b42a6f638fcde93"} Nov 02 22:32:39.572514 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.572210 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b" event={"ID":"26990ef1-f003-4196-b347-9dbf761a25ec","Type":"ContainerDied","Data":"a18a16a3b4743be77815ae29a3b9a57e55cdf5d7abe266a06c0e7235b1d941d9"} Nov 02 22:32:39.572514 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.572234 5871 scope.go:122] "RemoveContainer" containerID="f0fd8931639cdd84e03479096472e1aef215c828eac645139b42a6f638fcde93" Nov 02 22:32:39.583571 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.583544 5871 scope.go:122] "RemoveContainer" containerID="f0fd8931639cdd84e03479096472e1aef215c828eac645139b42a6f638fcde93" Nov 02 22:32:39.583930 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:32:39.583905 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0fd8931639cdd84e03479096472e1aef215c828eac645139b42a6f638fcde93\": container with ID starting with f0fd8931639cdd84e03479096472e1aef215c828eac645139b42a6f638fcde93 not found: ID does not exist" containerID="f0fd8931639cdd84e03479096472e1aef215c828eac645139b42a6f638fcde93" Nov 02 22:32:39.584005 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.583946 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0fd8931639cdd84e03479096472e1aef215c828eac645139b42a6f638fcde93"} err="failed to get container status \"f0fd8931639cdd84e03479096472e1aef215c828eac645139b42a6f638fcde93\": rpc error: code = NotFound desc = could not find container \"f0fd8931639cdd84e03479096472e1aef215c828eac645139b42a6f638fcde93\": container with ID starting with f0fd8931639cdd84e03479096472e1aef215c828eac645139b42a6f638fcde93 not found: ID does not exist" Nov 02 22:32:39.742461 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.742424 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b"] Nov 02 22:32:39.791328 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:39.791297 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2663d62e-dc8b888cd-wn68b"] Nov 02 22:32:40.362336 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.362293 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c"] Nov 02 22:32:40.362852 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.362535 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" podUID="23563615-f0f4-4cb4-9159-815000b8fa24" containerName="maas-api" containerID="cri-o://19308e84d0e0bb0ba8b3e6b45e6bf5839356f8bc3cab948d8f673d5d0066a3f8" gracePeriod=30 Nov 02 22:32:40.499865 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.499831 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:40.614964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.614842 5871 generic.go:356] "Generic (PLEG): container finished" podID="23563615-f0f4-4cb4-9159-815000b8fa24" containerID="19308e84d0e0bb0ba8b3e6b45e6bf5839356f8bc3cab948d8f673d5d0066a3f8" exitCode=0 Nov 02 22:32:40.615213 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.615189 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" event={"ID":"23563615-f0f4-4cb4-9159-815000b8fa24","Type":"ContainerDied","Data":"19308e84d0e0bb0ba8b3e6b45e6bf5839356f8bc3cab948d8f673d5d0066a3f8"} Nov 02 22:32:40.688588 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.688554 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:40.691346 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.691327 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:32:40.711358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.711319 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9"] Nov 02 22:32:40.745458 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.745426 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-maas-api-tls\") pod \"23563615-f0f4-4cb4-9159-815000b8fa24\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " Nov 02 22:32:40.745646 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.745473 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-metrics-certs\" (UniqueName: \"kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-metrics-certs\") pod \"23563615-f0f4-4cb4-9159-815000b8fa24\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " Nov 02 22:32:40.745646 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.745507 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23563615-f0f4-4cb4-9159-815000b8fa24-kube-api-access-lhspx\" (UniqueName: \"kubernetes.io/projected/23563615-f0f4-4cb4-9159-815000b8fa24-kube-api-access-lhspx\") pod \"23563615-f0f4-4cb4-9159-815000b8fa24\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " Nov 02 22:32:40.745769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.745740 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23563615-f0f4-4cb4-9159-815000b8fa24-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23563615-f0f4-4cb4-9159-815000b8fa24-openshift-service-ca\") pod \"23563615-f0f4-4cb4-9159-815000b8fa24\" (UID: \"23563615-f0f4-4cb4-9159-815000b8fa24\") " Nov 02 22:32:40.746120 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.746094 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23563615-f0f4-4cb4-9159-815000b8fa24-openshift-service-ca" pod "23563615-f0f4-4cb4-9159-815000b8fa24" (UID: "23563615-f0f4-4cb4-9159-815000b8fa24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:40.746321 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.746302 5871 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23563615-f0f4-4cb4-9159-815000b8fa24-openshift-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:40.747774 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.747744 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23563615-f0f4-4cb4-9159-815000b8fa24-kube-api-access-lhspx" pod "23563615-f0f4-4cb4-9159-815000b8fa24" (UID: "23563615-f0f4-4cb4-9159-815000b8fa24"). InnerVolumeSpecName "kube-api-access-lhspx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:40.747774 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.747766 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-metrics-certs" pod "23563615-f0f4-4cb4-9159-815000b8fa24" (UID: "23563615-f0f4-4cb4-9159-815000b8fa24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:40.747952 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.747785 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-maas-api-tls" pod "23563615-f0f4-4cb4-9159-815000b8fa24" (UID: "23563615-f0f4-4cb4-9159-815000b8fa24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:40.848055 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.848015 5871 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-maas-api-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:40.848055 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.848051 5871 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23563615-f0f4-4cb4-9159-815000b8fa24-metrics-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:40.848273 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.848062 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhspx\" (UniqueName: \"kubernetes.io/projected/23563615-f0f4-4cb4-9159-815000b8fa24-kube-api-access-lhspx\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:40.864343 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:40.864309 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:41.045435 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:41.045326 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:41.315814 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:41.315732 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26990ef1-f003-4196-b347-9dbf761a25ec" path="/var/lib/kubelet/pods/26990ef1-f003-4196-b347-9dbf761a25ec/volumes" Nov 02 22:32:41.456977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:41.456858 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:41.630283 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:41.630239 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" event={"ID":"23563615-f0f4-4cb4-9159-815000b8fa24","Type":"ContainerDied","Data":"9fcc6b298849e30dd8e9c8abb16d7dc9a17473ae8f07e78fc34aaf23a2e9e0b4"} Nov 02 22:32:41.630572 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:41.630553 5871 scope.go:122] "RemoveContainer" containerID="19308e84d0e0bb0ba8b3e6b45e6bf5839356f8bc3cab948d8f673d5d0066a3f8" Nov 02 22:32:41.630917 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:41.630848 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c" Nov 02 22:32:41.792944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:41.792905 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp"] Nov 02 22:32:41.796291 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:41.796244 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" podUID="20f6517a-1f4a-4eb9-b7cc-4b02d76cacae" containerName="maas-api" containerID="cri-o://11488850587f339b5e46d00fe743c44f64a6068c88e13e52dda04bbd370fcaac" gracePeriod=30 Nov 02 22:32:41.800954 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:41.800849 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c"] Nov 02 22:32:41.856660 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:41.856619 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3224ea-78cfb6ddcb-cv92c"] Nov 02 22:32:41.923185 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:41.923141 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:42.106353 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.106320 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:42.149817 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.149782 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:32:42.170447 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.170363 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-kube-api-access-lg825\" (UniqueName: \"kubernetes.io/projected/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-kube-api-access-lg825\") pod \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " Nov 02 22:32:42.170447 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.170403 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-metrics-certs\") pod \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " Nov 02 22:32:42.170702 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.170570 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-maas-api-tls\") pod \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " Nov 02 22:32:42.170702 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.170628 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-openshift-service-ca\") pod \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\" (UID: \"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae\") " Nov 02 22:32:42.171647 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.171618 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-openshift-service-ca" pod "20f6517a-1f4a-4eb9-b7cc-4b02d76cacae" (UID: "20f6517a-1f4a-4eb9-b7cc-4b02d76cacae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:42.179788 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.179748 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-metrics-certs" pod "20f6517a-1f4a-4eb9-b7cc-4b02d76cacae" (UID: "20f6517a-1f4a-4eb9-b7cc-4b02d76cacae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:42.181440 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.181406 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-kube-api-access-lg825" pod "20f6517a-1f4a-4eb9-b7cc-4b02d76cacae" (UID: "20f6517a-1f4a-4eb9-b7cc-4b02d76cacae"). InnerVolumeSpecName "kube-api-access-lg825". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:42.190236 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.190184 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-maas-api-tls" pod "20f6517a-1f4a-4eb9-b7cc-4b02d76cacae" (UID: "20f6517a-1f4a-4eb9-b7cc-4b02d76cacae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:42.272733 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.272697 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg825\" (UniqueName: \"kubernetes.io/projected/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-kube-api-access-lg825\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:42.272733 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.272728 5871 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-metrics-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:42.272733 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.272741 5871 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-maas-api-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:42.273115 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.272757 5871 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae-openshift-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:42.291004 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.288578 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:42.572435 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.572337 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:42.642382 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.642340 5871 generic.go:356] "Generic (PLEG): container finished" podID="20f6517a-1f4a-4eb9-b7cc-4b02d76cacae" containerID="11488850587f339b5e46d00fe743c44f64a6068c88e13e52dda04bbd370fcaac" exitCode=0 Nov 02 22:32:42.642586 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.642440 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" event={"ID":"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae","Type":"ContainerDied","Data":"11488850587f339b5e46d00fe743c44f64a6068c88e13e52dda04bbd370fcaac"} Nov 02 22:32:42.642586 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.642514 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" event={"ID":"20f6517a-1f4a-4eb9-b7cc-4b02d76cacae","Type":"ContainerDied","Data":"0ede54770d2f36afb59f81fb3736c6d0ec16c1d98f1b0755f32e1690ab17fd26"} Nov 02 22:32:42.642586 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.642546 5871 scope.go:122] "RemoveContainer" containerID="11488850587f339b5e46d00fe743c44f64a6068c88e13e52dda04bbd370fcaac" Nov 02 22:32:42.642824 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.642794 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp" Nov 02 22:32:42.655057 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.655030 5871 scope.go:122] "RemoveContainer" containerID="11488850587f339b5e46d00fe743c44f64a6068c88e13e52dda04bbd370fcaac" Nov 02 22:32:42.655416 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:32:42.655393 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11488850587f339b5e46d00fe743c44f64a6068c88e13e52dda04bbd370fcaac\": container with ID starting with 11488850587f339b5e46d00fe743c44f64a6068c88e13e52dda04bbd370fcaac not found: ID does not exist" containerID="11488850587f339b5e46d00fe743c44f64a6068c88e13e52dda04bbd370fcaac" Nov 02 22:32:42.655484 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.655428 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11488850587f339b5e46d00fe743c44f64a6068c88e13e52dda04bbd370fcaac"} err="failed to get container status \"11488850587f339b5e46d00fe743c44f64a6068c88e13e52dda04bbd370fcaac\": rpc error: code = NotFound desc = could not find container \"11488850587f339b5e46d00fe743c44f64a6068c88e13e52dda04bbd370fcaac\": container with ID starting with 11488850587f339b5e46d00fe743c44f64a6068c88e13e52dda04bbd370fcaac not found: ID does not exist" Nov 02 22:32:42.811645 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.811608 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp"] Nov 02 22:32:42.894233 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:42.894188 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb670683-6dc5b77-p2qxp"] Nov 02 22:32:43.292702 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:43.292602 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:43.315858 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:43.315820 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f6517a-1f4a-4eb9-b7cc-4b02d76cacae" path="/var/lib/kubelet/pods/20f6517a-1f4a-4eb9-b7cc-4b02d76cacae/volumes" Nov 02 22:32:43.316421 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:43.316405 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23563615-f0f4-4cb4-9159-815000b8fa24" path="/var/lib/kubelet/pods/23563615-f0f4-4cb4-9159-815000b8fa24/volumes" Nov 02 22:32:43.772584 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:32:43.772548 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:32:43.939866 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:43.939836 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:44.110475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:44.110365 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:44.297176 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:44.297142 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:45.149208 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:45.149165 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:45.643750 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:45.643717 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:45.849082 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:45.849051 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:46.881175 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:46.881136 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k"] Nov 02 22:32:47.004200 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:47.004167 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:47.483623 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:47.483585 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:47.651646 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:47.651608 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:49.285673 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:49.285644 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:49.458092 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:49.458056 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:49.630461 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:49.630430 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6"] Nov 02 22:32:49.630656 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:49.630634 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" podUID="f2d38074-6059-4106-96a0-46fff0b1663c" containerName="istio-proxy" containerID="cri-o://a400ae5a3d4b1c1102bf86484bf49d3f02dac1f293d9e47aff81976e48878abb" gracePeriod=30 Nov 02 22:32:49.644117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:49.644062 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:49.648483 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:49.648449 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh"] Nov 02 22:32:49.648819 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:49.648789 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" podUID="4839ef24-050d-41ea-95c3-7713d6df39a0" containerName="istio-proxy" containerID="cri-o://b908cb8162fd2b56349ad3a8178033c2aee7c22ad28c7108888f647e68bd2b08" gracePeriod=30 Nov 02 22:32:49.672757 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:49.672723 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8"] Nov 02 22:32:49.673026 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:49.672989 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" podUID="3210ef32-d6e6-47d3-b431-826aba457e12" containerName="istio-proxy" containerID="cri-o://2897bee7bd40bef91fee1c27ee963b886ccf2a9a8eb068b7a5176cd3f4c1d656" gracePeriod=30 Nov 02 22:32:49.820225 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:49.820186 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:50.088068 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:50.087987 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:32:50.088068 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:50.088051 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" podUID="4839ef24-050d-41ea-95c3-7713d6df39a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:32:50.240946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:50.240905 5871 patch_prober.go:29] interesting pod/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:32:50.241121 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:50.240966 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" podUID="3210ef32-d6e6-47d3-b431-826aba457e12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:32:50.706533 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:50.706492 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq"] Nov 02 22:32:50.715402 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:50.715370 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb"] Nov 02 22:32:50.740754 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:50.740708 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5"] Nov 02 22:32:50.898382 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:50.898350 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:54.748327 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:54.748293 5871 generic.go:356] "Generic (PLEG): container finished" podID="f2d38074-6059-4106-96a0-46fff0b1663c" containerID="a400ae5a3d4b1c1102bf86484bf49d3f02dac1f293d9e47aff81976e48878abb" exitCode=0 Nov 02 22:32:54.748700 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:54.748360 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" event={"ID":"f2d38074-6059-4106-96a0-46fff0b1663c","Type":"ContainerDied","Data":"a400ae5a3d4b1c1102bf86484bf49d3f02dac1f293d9e47aff81976e48878abb"} Nov 02 22:32:54.797009 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:54.796972 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:55.018858 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.018824 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:32:55.040151 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.040121 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:32:55.059772 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.059739 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:32:55.076952 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.076826 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d38074-6059-4106-96a0-46fff0b1663c-kube-api-access-85rlx\" (UniqueName: \"kubernetes.io/projected/f2d38074-6059-4106-96a0-46fff0b1663c-kube-api-access-85rlx\") pod \"f2d38074-6059-4106-96a0-46fff0b1663c\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " Nov 02 22:32:55.077186 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.076986 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d38074-6059-4106-96a0-46fff0b1663c-istio-token\" (UniqueName: \"kubernetes.io/projected/f2d38074-6059-4106-96a0-46fff0b1663c-istio-token\") pod \"f2d38074-6059-4106-96a0-46fff0b1663c\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " Nov 02 22:32:55.077186 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.077044 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-workload-certs\") pod \"f2d38074-6059-4106-96a0-46fff0b1663c\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " Nov 02 22:32:55.077186 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.077143 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-credential-socket\") pod \"f2d38074-6059-4106-96a0-46fff0b1663c\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " Nov 02 22:32:55.077364 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.077222 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2d38074-6059-4106-96a0-46fff0b1663c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d38074-6059-4106-96a0-46fff0b1663c-istio-podinfo\") pod \"f2d38074-6059-4106-96a0-46fff0b1663c\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " Nov 02 22:32:55.077415 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.077393 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-istio-envoy\") pod \"f2d38074-6059-4106-96a0-46fff0b1663c\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " Nov 02 22:32:55.077479 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.077449 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-workload-socket\") pod \"f2d38074-6059-4106-96a0-46fff0b1663c\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " Nov 02 22:32:55.077532 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.077480 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-istio-data\") pod \"f2d38074-6059-4106-96a0-46fff0b1663c\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " Nov 02 22:32:55.077595 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.077577 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2d38074-6059-4106-96a0-46fff0b1663c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d38074-6059-4106-96a0-46fff0b1663c-istiod-ca-cert\") pod \"f2d38074-6059-4106-96a0-46fff0b1663c\" (UID: \"f2d38074-6059-4106-96a0-46fff0b1663c\") " Nov 02 22:32:55.078582 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.078512 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-credential-socket" pod "f2d38074-6059-4106-96a0-46fff0b1663c" (UID: "f2d38074-6059-4106-96a0-46fff0b1663c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.078693 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.078626 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-workload-socket" pod "f2d38074-6059-4106-96a0-46fff0b1663c" (UID: "f2d38074-6059-4106-96a0-46fff0b1663c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.078772 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.078703 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-istio-data" pod "f2d38074-6059-4106-96a0-46fff0b1663c" (UID: "f2d38074-6059-4106-96a0-46fff0b1663c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.079137 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.079112 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d38074-6059-4106-96a0-46fff0b1663c-istiod-ca-cert" pod "f2d38074-6059-4106-96a0-46fff0b1663c" (UID: "f2d38074-6059-4106-96a0-46fff0b1663c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:55.079396 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.079375 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.079467 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.079403 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.079467 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.079419 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d38074-6059-4106-96a0-46fff0b1663c-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.079467 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.079436 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.079724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.079703 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-workload-certs" pod "f2d38074-6059-4106-96a0-46fff0b1663c" (UID: "f2d38074-6059-4106-96a0-46fff0b1663c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.081236 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.081211 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2d38074-6059-4106-96a0-46fff0b1663c-istio-podinfo" pod "f2d38074-6059-4106-96a0-46fff0b1663c" (UID: "f2d38074-6059-4106-96a0-46fff0b1663c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:32:55.083813 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.083203 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d38074-6059-4106-96a0-46fff0b1663c-kube-api-access-85rlx" pod "f2d38074-6059-4106-96a0-46fff0b1663c" (UID: "f2d38074-6059-4106-96a0-46fff0b1663c"). InnerVolumeSpecName "kube-api-access-85rlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:55.083813 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.083776 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-istio-envoy" pod "f2d38074-6059-4106-96a0-46fff0b1663c" (UID: "f2d38074-6059-4106-96a0-46fff0b1663c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.085929 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.085902 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d38074-6059-4106-96a0-46fff0b1663c-istio-token" pod "f2d38074-6059-4106-96a0-46fff0b1663c" (UID: "f2d38074-6059-4106-96a0-46fff0b1663c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:55.164428 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.164394 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:55.180179 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.180136 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-podinfo\") pod \"4839ef24-050d-41ea-95c3-7713d6df39a0\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " Nov 02 22:32:55.180365 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.180220 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3210ef32-d6e6-47d3-b431-826aba457e12-kube-api-access-wlbzf\" (UniqueName: \"kubernetes.io/projected/3210ef32-d6e6-47d3-b431-826aba457e12-kube-api-access-wlbzf\") pod \"3210ef32-d6e6-47d3-b431-826aba457e12\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " Nov 02 22:32:55.180365 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.180282 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3210ef32-d6e6-47d3-b431-826aba457e12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3210ef32-d6e6-47d3-b431-826aba457e12-istiod-ca-cert\") pod \"3210ef32-d6e6-47d3-b431-826aba457e12\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " Nov 02 22:32:55.180365 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.180321 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-credential-socket\") pod \"4839ef24-050d-41ea-95c3-7713d6df39a0\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " Nov 02 22:32:55.180365 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.180344 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-workload-socket\") pod \"4839ef24-050d-41ea-95c3-7713d6df39a0\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " Nov 02 22:32:55.180586 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.180381 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4839ef24-050d-41ea-95c3-7713d6df39a0-kube-api-access-ct5hl\" (UniqueName: \"kubernetes.io/projected/4839ef24-050d-41ea-95c3-7713d6df39a0-kube-api-access-ct5hl\") pod \"4839ef24-050d-41ea-95c3-7713d6df39a0\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " Nov 02 22:32:55.180586 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.180425 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4839ef24-050d-41ea-95c3-7713d6df39a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4839ef24-050d-41ea-95c3-7713d6df39a0-istiod-ca-cert\") pod \"4839ef24-050d-41ea-95c3-7713d6df39a0\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " Nov 02 22:32:55.180586 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.180481 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-data\") pod \"4839ef24-050d-41ea-95c3-7713d6df39a0\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " Nov 02 22:32:55.180827 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.180592 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3210ef32-d6e6-47d3-b431-826aba457e12-istio-token\" (UniqueName: \"kubernetes.io/projected/3210ef32-d6e6-47d3-b431-826aba457e12-istio-token\") pod \"3210ef32-d6e6-47d3-b431-826aba457e12\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " Nov 02 22:32:55.180827 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.180656 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-istio-envoy\") pod \"3210ef32-d6e6-47d3-b431-826aba457e12\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " Nov 02 22:32:55.180827 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.180726 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-token\" (UniqueName: \"kubernetes.io/projected/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-token\") pod \"4839ef24-050d-41ea-95c3-7713d6df39a0\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " Nov 02 22:32:55.180827 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.180754 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-workload-certs\") pod \"4839ef24-050d-41ea-95c3-7713d6df39a0\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " Nov 02 22:32:55.180827 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.180785 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-credential-socket\") pod \"3210ef32-d6e6-47d3-b431-826aba457e12\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " Nov 02 22:32:55.181108 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.180837 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-workload-certs\") pod \"3210ef32-d6e6-47d3-b431-826aba457e12\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " Nov 02 22:32:55.181108 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.180912 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3210ef32-d6e6-47d3-b431-826aba457e12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3210ef32-d6e6-47d3-b431-826aba457e12-istio-podinfo\") pod \"3210ef32-d6e6-47d3-b431-826aba457e12\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " Nov 02 22:32:55.181108 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.180952 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-envoy\") pod \"4839ef24-050d-41ea-95c3-7713d6df39a0\" (UID: \"4839ef24-050d-41ea-95c3-7713d6df39a0\") " Nov 02 22:32:55.181108 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.181067 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-istio-data\") pod \"3210ef32-d6e6-47d3-b431-826aba457e12\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " Nov 02 22:32:55.181352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.181117 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-workload-socket\") pod \"3210ef32-d6e6-47d3-b431-826aba457e12\" (UID: \"3210ef32-d6e6-47d3-b431-826aba457e12\") " Nov 02 22:32:55.181733 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.181711 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85rlx\" (UniqueName: \"kubernetes.io/projected/f2d38074-6059-4106-96a0-46fff0b1663c-kube-api-access-85rlx\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.181822 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.181738 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d38074-6059-4106-96a0-46fff0b1663c-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.181822 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.181749 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.181822 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.181757 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d38074-6059-4106-96a0-46fff0b1663c-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.181822 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.181766 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d38074-6059-4106-96a0-46fff0b1663c-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.182120 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.182027 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-workload-socket" pod "3210ef32-d6e6-47d3-b431-826aba457e12" (UID: "3210ef32-d6e6-47d3-b431-826aba457e12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.183095 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.182767 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-credential-socket" pod "4839ef24-050d-41ea-95c3-7713d6df39a0" (UID: "4839ef24-050d-41ea-95c3-7713d6df39a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.183219 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.183185 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3210ef32-d6e6-47d3-b431-826aba457e12-istiod-ca-cert" pod "3210ef32-d6e6-47d3-b431-826aba457e12" (UID: "3210ef32-d6e6-47d3-b431-826aba457e12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:55.183730 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.183675 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-workload-socket" pod "4839ef24-050d-41ea-95c3-7713d6df39a0" (UID: "4839ef24-050d-41ea-95c3-7713d6df39a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.183825 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.183764 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-workload-certs" pod "3210ef32-d6e6-47d3-b431-826aba457e12" (UID: "3210ef32-d6e6-47d3-b431-826aba457e12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.183825 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.183796 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-workload-certs" pod "4839ef24-050d-41ea-95c3-7713d6df39a0" (UID: "4839ef24-050d-41ea-95c3-7713d6df39a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.184159 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.183984 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-credential-socket" pod "3210ef32-d6e6-47d3-b431-826aba457e12" (UID: "3210ef32-d6e6-47d3-b431-826aba457e12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.184778 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.184202 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-istio-data" pod "3210ef32-d6e6-47d3-b431-826aba457e12" (UID: "3210ef32-d6e6-47d3-b431-826aba457e12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.184778 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.184556 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-data" pod "4839ef24-050d-41ea-95c3-7713d6df39a0" (UID: "4839ef24-050d-41ea-95c3-7713d6df39a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.185152 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.185112 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4839ef24-050d-41ea-95c3-7713d6df39a0-istiod-ca-cert" pod "4839ef24-050d-41ea-95c3-7713d6df39a0" (UID: "4839ef24-050d-41ea-95c3-7713d6df39a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:55.185968 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.185940 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-podinfo" pod "4839ef24-050d-41ea-95c3-7713d6df39a0" (UID: "4839ef24-050d-41ea-95c3-7713d6df39a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:32:55.202908 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.202815 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-istio-envoy" pod "3210ef32-d6e6-47d3-b431-826aba457e12" (UID: "3210ef32-d6e6-47d3-b431-826aba457e12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.202908 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.202819 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3210ef32-d6e6-47d3-b431-826aba457e12-kube-api-access-wlbzf" pod "3210ef32-d6e6-47d3-b431-826aba457e12" (UID: "3210ef32-d6e6-47d3-b431-826aba457e12"). InnerVolumeSpecName "kube-api-access-wlbzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:55.202908 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.202820 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3210ef32-d6e6-47d3-b431-826aba457e12-istio-podinfo" pod "3210ef32-d6e6-47d3-b431-826aba457e12" (UID: "3210ef32-d6e6-47d3-b431-826aba457e12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:32:55.203149 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.202971 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-token" pod "4839ef24-050d-41ea-95c3-7713d6df39a0" (UID: "4839ef24-050d-41ea-95c3-7713d6df39a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:55.203149 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.202978 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4839ef24-050d-41ea-95c3-7713d6df39a0-kube-api-access-ct5hl" pod "4839ef24-050d-41ea-95c3-7713d6df39a0" (UID: "4839ef24-050d-41ea-95c3-7713d6df39a0"). InnerVolumeSpecName "kube-api-access-ct5hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:55.203261 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.203239 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3210ef32-d6e6-47d3-b431-826aba457e12-istio-token" pod "3210ef32-d6e6-47d3-b431-826aba457e12" (UID: "3210ef32-d6e6-47d3-b431-826aba457e12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:55.204559 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.204465 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-envoy" pod "4839ef24-050d-41ea-95c3-7713d6df39a0" (UID: "4839ef24-050d-41ea-95c3-7713d6df39a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:55.284179 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.284139 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlbzf\" (UniqueName: \"kubernetes.io/projected/3210ef32-d6e6-47d3-b431-826aba457e12-kube-api-access-wlbzf\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.284430 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.284409 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3210ef32-d6e6-47d3-b431-826aba457e12-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.284574 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.284540 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.284679 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.284663 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.284765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.284752 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct5hl\" (UniqueName: \"kubernetes.io/projected/4839ef24-050d-41ea-95c3-7713d6df39a0-kube-api-access-ct5hl\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.284861 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.284849 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4839ef24-050d-41ea-95c3-7713d6df39a0-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.285012 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.284994 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.285164 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.285149 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3210ef32-d6e6-47d3-b431-826aba457e12-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.285294 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.285264 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.285395 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.285380 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.285490 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.285477 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.285591 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.285576 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.285679 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.285667 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.285771 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.285758 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3210ef32-d6e6-47d3-b431-826aba457e12-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.285859 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.285847 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.285983 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.285970 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.286077 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.286064 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3210ef32-d6e6-47d3-b431-826aba457e12-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.286160 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.286148 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4839ef24-050d-41ea-95c3-7713d6df39a0-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:55.346364 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.345018 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:55.742610 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.742568 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" containerID="cri-o://483e2bd3c86e6a5cea18df1bef6197d4cb5da501332dac72c64310059c0a7c07" gracePeriod=45 Nov 02 22:32:55.758343 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.758309 5871 generic.go:356] "Generic (PLEG): container finished" podID="4839ef24-050d-41ea-95c3-7713d6df39a0" containerID="b908cb8162fd2b56349ad3a8178033c2aee7c22ad28c7108888f647e68bd2b08" exitCode=0 Nov 02 22:32:55.758780 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.758356 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" Nov 02 22:32:55.758780 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.758382 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" event={"ID":"4839ef24-050d-41ea-95c3-7713d6df39a0","Type":"ContainerDied","Data":"b908cb8162fd2b56349ad3a8178033c2aee7c22ad28c7108888f647e68bd2b08"} Nov 02 22:32:55.758780 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.758437 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh" event={"ID":"4839ef24-050d-41ea-95c3-7713d6df39a0","Type":"ContainerDied","Data":"5ba4ae88cdcb50cf8c94528726a7493674bf506f0aaac726f89fdd222d34b4f1"} Nov 02 22:32:55.758780 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.758468 5871 scope.go:122] "RemoveContainer" containerID="b908cb8162fd2b56349ad3a8178033c2aee7c22ad28c7108888f647e68bd2b08" Nov 02 22:32:55.760041 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.760022 5871 generic.go:356] "Generic (PLEG): container finished" podID="3210ef32-d6e6-47d3-b431-826aba457e12" containerID="2897bee7bd40bef91fee1c27ee963b886ccf2a9a8eb068b7a5176cd3f4c1d656" exitCode=0 Nov 02 22:32:55.760118 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.760076 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" Nov 02 22:32:55.760171 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.760147 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" event={"ID":"3210ef32-d6e6-47d3-b431-826aba457e12","Type":"ContainerDied","Data":"2897bee7bd40bef91fee1c27ee963b886ccf2a9a8eb068b7a5176cd3f4c1d656"} Nov 02 22:32:55.760242 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.760203 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8" event={"ID":"3210ef32-d6e6-47d3-b431-826aba457e12","Type":"ContainerDied","Data":"6192ea125c940e65b358b70a100ab588b1edd1981d27474e003b01194eb3f1dc"} Nov 02 22:32:55.762306 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.762279 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" event={"ID":"f2d38074-6059-4106-96a0-46fff0b1663c","Type":"ContainerDied","Data":"f95326547f8270c1846922b7da88058a90bf9398d5efdcbece7d7ffce0722e67"} Nov 02 22:32:55.762420 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.762352 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6" Nov 02 22:32:55.856322 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.856135 5871 scope.go:122] "RemoveContainer" containerID="615c7c1e55c9aaf2754a2fcef8f78957d7abab0d09e57361dc6ca9e916b58d2c" Nov 02 22:32:55.866332 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.866309 5871 scope.go:122] "RemoveContainer" containerID="b908cb8162fd2b56349ad3a8178033c2aee7c22ad28c7108888f647e68bd2b08" Nov 02 22:32:55.866671 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:32:55.866648 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b908cb8162fd2b56349ad3a8178033c2aee7c22ad28c7108888f647e68bd2b08\": container with ID starting with b908cb8162fd2b56349ad3a8178033c2aee7c22ad28c7108888f647e68bd2b08 not found: ID does not exist" containerID="b908cb8162fd2b56349ad3a8178033c2aee7c22ad28c7108888f647e68bd2b08" Nov 02 22:32:55.866742 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.866681 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b908cb8162fd2b56349ad3a8178033c2aee7c22ad28c7108888f647e68bd2b08"} err="failed to get container status \"b908cb8162fd2b56349ad3a8178033c2aee7c22ad28c7108888f647e68bd2b08\": rpc error: code = NotFound desc = could not find container \"b908cb8162fd2b56349ad3a8178033c2aee7c22ad28c7108888f647e68bd2b08\": container with ID starting with b908cb8162fd2b56349ad3a8178033c2aee7c22ad28c7108888f647e68bd2b08 not found: ID does not exist" Nov 02 22:32:55.866742 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.866705 5871 scope.go:122] "RemoveContainer" containerID="615c7c1e55c9aaf2754a2fcef8f78957d7abab0d09e57361dc6ca9e916b58d2c" Nov 02 22:32:55.866990 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:32:55.866969 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615c7c1e55c9aaf2754a2fcef8f78957d7abab0d09e57361dc6ca9e916b58d2c\": container with ID starting with 615c7c1e55c9aaf2754a2fcef8f78957d7abab0d09e57361dc6ca9e916b58d2c not found: ID does not exist" containerID="615c7c1e55c9aaf2754a2fcef8f78957d7abab0d09e57361dc6ca9e916b58d2c" Nov 02 22:32:55.867037 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.866998 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615c7c1e55c9aaf2754a2fcef8f78957d7abab0d09e57361dc6ca9e916b58d2c"} err="failed to get container status \"615c7c1e55c9aaf2754a2fcef8f78957d7abab0d09e57361dc6ca9e916b58d2c\": rpc error: code = NotFound desc = could not find container \"615c7c1e55c9aaf2754a2fcef8f78957d7abab0d09e57361dc6ca9e916b58d2c\": container with ID starting with 615c7c1e55c9aaf2754a2fcef8f78957d7abab0d09e57361dc6ca9e916b58d2c not found: ID does not exist" Nov 02 22:32:55.867037 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.867018 5871 scope.go:122] "RemoveContainer" containerID="2897bee7bd40bef91fee1c27ee963b886ccf2a9a8eb068b7a5176cd3f4c1d656" Nov 02 22:32:55.878058 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.878021 5871 scope.go:122] "RemoveContainer" containerID="2897bee7bd40bef91fee1c27ee963b886ccf2a9a8eb068b7a5176cd3f4c1d656" Nov 02 22:32:55.878430 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:32:55.878404 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2897bee7bd40bef91fee1c27ee963b886ccf2a9a8eb068b7a5176cd3f4c1d656\": container with ID starting with 2897bee7bd40bef91fee1c27ee963b886ccf2a9a8eb068b7a5176cd3f4c1d656 not found: ID does not exist" containerID="2897bee7bd40bef91fee1c27ee963b886ccf2a9a8eb068b7a5176cd3f4c1d656" Nov 02 22:32:55.878555 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.878440 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2897bee7bd40bef91fee1c27ee963b886ccf2a9a8eb068b7a5176cd3f4c1d656"} err="failed to get container status \"2897bee7bd40bef91fee1c27ee963b886ccf2a9a8eb068b7a5176cd3f4c1d656\": rpc error: code = NotFound desc = could not find container \"2897bee7bd40bef91fee1c27ee963b886ccf2a9a8eb068b7a5176cd3f4c1d656\": container with ID starting with 2897bee7bd40bef91fee1c27ee963b886ccf2a9a8eb068b7a5176cd3f4c1d656 not found: ID does not exist" Nov 02 22:32:55.878555 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:55.878469 5871 scope.go:122] "RemoveContainer" containerID="a400ae5a3d4b1c1102bf86484bf49d3f02dac1f293d9e47aff81976e48878abb" Nov 02 22:32:56.033395 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.033361 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8"] Nov 02 22:32:56.035863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.035846 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:32:56.082897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.082852 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-rjhc8"] Nov 02 22:32:56.098295 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.098256 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-tmp-dir\") pod \"8f259989-289c-4b0d-a158-9755a2ecb9fe\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " Nov 02 22:32:56.098475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.098349 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-model-cache\") pod \"8f259989-289c-4b0d-a158-9755a2ecb9fe\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " Nov 02 22:32:56.098475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.098421 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f259989-289c-4b0d-a158-9755a2ecb9fe-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f259989-289c-4b0d-a158-9755a2ecb9fe-tls-certs\") pod \"8f259989-289c-4b0d-a158-9755a2ecb9fe\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " Nov 02 22:32:56.098475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.098471 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-home\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-home\") pod \"8f259989-289c-4b0d-a158-9755a2ecb9fe\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " Nov 02 22:32:56.098643 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.098528 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-tmp-dir" pod "8f259989-289c-4b0d-a158-9755a2ecb9fe" (UID: "8f259989-289c-4b0d-a158-9755a2ecb9fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:56.098643 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.098555 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-dshm\") pod \"8f259989-289c-4b0d-a158-9755a2ecb9fe\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " Nov 02 22:32:56.098643 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.098632 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-model-cache" pod "8f259989-289c-4b0d-a158-9755a2ecb9fe" (UID: "8f259989-289c-4b0d-a158-9755a2ecb9fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:56.098754 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.098692 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f259989-289c-4b0d-a158-9755a2ecb9fe-kube-api-access-c95wk\" (UniqueName: \"kubernetes.io/projected/8f259989-289c-4b0d-a158-9755a2ecb9fe-kube-api-access-c95wk\") pod \"8f259989-289c-4b0d-a158-9755a2ecb9fe\" (UID: \"8f259989-289c-4b0d-a158-9755a2ecb9fe\") " Nov 02 22:32:56.098754 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.098728 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-home" pod "8f259989-289c-4b0d-a158-9755a2ecb9fe" (UID: "8f259989-289c-4b0d-a158-9755a2ecb9fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:56.099223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.099199 5871 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-tmp-dir\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:56.099223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.099220 5871 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-model-cache\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:56.099452 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.099232 5871 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-home\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:56.100852 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.100826 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-dshm" pod "8f259989-289c-4b0d-a158-9755a2ecb9fe" (UID: "8f259989-289c-4b0d-a158-9755a2ecb9fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:56.100959 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.100884 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f259989-289c-4b0d-a158-9755a2ecb9fe-kube-api-access-c95wk" pod "8f259989-289c-4b0d-a158-9755a2ecb9fe" (UID: "8f259989-289c-4b0d-a158-9755a2ecb9fe"). InnerVolumeSpecName "kube-api-access-c95wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:56.100959 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.100939 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f259989-289c-4b0d-a158-9755a2ecb9fe-tls-certs" pod "8f259989-289c-4b0d-a158-9755a2ecb9fe" (UID: "8f259989-289c-4b0d-a158-9755a2ecb9fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:56.158840 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.154423 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh"] Nov 02 22:32:56.158840 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.158280 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-g7wzh"] Nov 02 22:32:56.205467 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.203604 5871 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f259989-289c-4b0d-a158-9755a2ecb9fe-tls-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:56.205467 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.203635 5871 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f259989-289c-4b0d-a158-9755a2ecb9fe-dshm\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:56.205467 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.203647 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c95wk\" (UniqueName: \"kubernetes.io/projected/8f259989-289c-4b0d-a158-9755a2ecb9fe-kube-api-access-c95wk\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:56.360905 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.360503 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6"] Nov 02 22:32:56.360905 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.360575 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3224ea-openshift-default-7c9cb65579-7ksc6"] Nov 02 22:32:56.773239 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.773199 5871 generic.go:356] "Generic (PLEG): container finished" podID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerID="483e2bd3c86e6a5cea18df1bef6197d4cb5da501332dac72c64310059c0a7c07" exitCode=0 Nov 02 22:32:56.773729 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.773263 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" Nov 02 22:32:56.773729 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.773295 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" event={"ID":"8f259989-289c-4b0d-a158-9755a2ecb9fe","Type":"ContainerDied","Data":"483e2bd3c86e6a5cea18df1bef6197d4cb5da501332dac72c64310059c0a7c07"} Nov 02 22:32:56.773729 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.773346 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9" event={"ID":"8f259989-289c-4b0d-a158-9755a2ecb9fe","Type":"ContainerDied","Data":"431a648041e122bbcccf12b4be073a3ab120d30c9278a7f89ef3b95149b37187"} Nov 02 22:32:56.773729 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.773371 5871 scope.go:122] "RemoveContainer" containerID="483e2bd3c86e6a5cea18df1bef6197d4cb5da501332dac72c64310059c0a7c07" Nov 02 22:32:56.784482 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.784456 5871 scope.go:122] "RemoveContainer" containerID="483e2bd3c86e6a5cea18df1bef6197d4cb5da501332dac72c64310059c0a7c07" Nov 02 22:32:56.784839 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:32:56.784816 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"483e2bd3c86e6a5cea18df1bef6197d4cb5da501332dac72c64310059c0a7c07\": container with ID starting with 483e2bd3c86e6a5cea18df1bef6197d4cb5da501332dac72c64310059c0a7c07 not found: ID does not exist" containerID="483e2bd3c86e6a5cea18df1bef6197d4cb5da501332dac72c64310059c0a7c07" Nov 02 22:32:56.784976 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.784847 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"483e2bd3c86e6a5cea18df1bef6197d4cb5da501332dac72c64310059c0a7c07"} err="failed to get container status \"483e2bd3c86e6a5cea18df1bef6197d4cb5da501332dac72c64310059c0a7c07\": rpc error: code = NotFound desc = could not find container \"483e2bd3c86e6a5cea18df1bef6197d4cb5da501332dac72c64310059c0a7c07\": container with ID starting with 483e2bd3c86e6a5cea18df1bef6197d4cb5da501332dac72c64310059c0a7c07 not found: ID does not exist" Nov 02 22:32:56.942483 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.942453 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9"] Nov 02 22:32:56.945799 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:56.945772 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-49a557/premium-w6-49a557-kserve-8f69bfbcf-644d9"] Nov 02 22:32:57.314667 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:57.314627 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3210ef32-d6e6-47d3-b431-826aba457e12" path="/var/lib/kubelet/pods/3210ef32-d6e6-47d3-b431-826aba457e12/volumes" Nov 02 22:32:57.315209 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:57.315195 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4839ef24-050d-41ea-95c3-7713d6df39a0" path="/var/lib/kubelet/pods/4839ef24-050d-41ea-95c3-7713d6df39a0/volumes" Nov 02 22:32:57.315622 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:57.315610 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" path="/var/lib/kubelet/pods/8f259989-289c-4b0d-a158-9755a2ecb9fe/volumes" Nov 02 22:32:57.316399 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:57.316384 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2d38074-6059-4106-96a0-46fff0b1663c" path="/var/lib/kubelet/pods/f2d38074-6059-4106-96a0-46fff0b1663c/volumes" Nov 02 22:32:57.519141 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:57.519099 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:57.706247 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:57.704962 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:32:59.938262 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:59.938225 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2"] Nov 02 22:32:59.939065 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:32:59.939023 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" podUID="fd88af66-9693-4cd1-abd5-d2af89caa020" containerName="maas-api" containerID="cri-o://c0bf40f63c8a02aced0aa4292f4d2d059e671c5df724c928cef3a0eea59d964c" gracePeriod=30 Nov 02 22:33:00.072231 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.072201 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:00.258264 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.258234 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:33:00.373063 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.373020 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd88af66-9693-4cd1-abd5-d2af89caa020-kube-api-access-wlm6l\" (UniqueName: \"kubernetes.io/projected/fd88af66-9693-4cd1-abd5-d2af89caa020-kube-api-access-wlm6l\") pod \"fd88af66-9693-4cd1-abd5-d2af89caa020\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " Nov 02 22:33:00.373308 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.373144 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-maas-api-tls\") pod \"fd88af66-9693-4cd1-abd5-d2af89caa020\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " Nov 02 22:33:00.373788 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.373731 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd88af66-9693-4cd1-abd5-d2af89caa020-openshift-service-ca" pod "fd88af66-9693-4cd1-abd5-d2af89caa020" (UID: "fd88af66-9693-4cd1-abd5-d2af89caa020"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:00.373989 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.373805 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd88af66-9693-4cd1-abd5-d2af89caa020-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd88af66-9693-4cd1-abd5-d2af89caa020-openshift-service-ca\") pod \"fd88af66-9693-4cd1-abd5-d2af89caa020\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " Nov 02 22:33:00.373989 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.373936 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-metrics-certs\") pod \"fd88af66-9693-4cd1-abd5-d2af89caa020\" (UID: \"fd88af66-9693-4cd1-abd5-d2af89caa020\") " Nov 02 22:33:00.374712 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.374684 5871 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd88af66-9693-4cd1-abd5-d2af89caa020-openshift-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:00.375847 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.375814 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-maas-api-tls" pod "fd88af66-9693-4cd1-abd5-d2af89caa020" (UID: "fd88af66-9693-4cd1-abd5-d2af89caa020"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:00.376491 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.376459 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd88af66-9693-4cd1-abd5-d2af89caa020-kube-api-access-wlm6l" pod "fd88af66-9693-4cd1-abd5-d2af89caa020" (UID: "fd88af66-9693-4cd1-abd5-d2af89caa020"). InnerVolumeSpecName "kube-api-access-wlm6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:00.376592 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.376518 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-metrics-certs" pod "fd88af66-9693-4cd1-abd5-d2af89caa020" (UID: "fd88af66-9693-4cd1-abd5-d2af89caa020"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:00.477824 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.477721 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlm6l\" (UniqueName: \"kubernetes.io/projected/fd88af66-9693-4cd1-abd5-d2af89caa020-kube-api-access-wlm6l\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:00.477824 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.477764 5871 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-maas-api-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:00.477824 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.477779 5871 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd88af66-9693-4cd1-abd5-d2af89caa020-metrics-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:00.807692 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.807582 5871 generic.go:356] "Generic (PLEG): container finished" podID="fd88af66-9693-4cd1-abd5-d2af89caa020" containerID="c0bf40f63c8a02aced0aa4292f4d2d059e671c5df724c928cef3a0eea59d964c" exitCode=0 Nov 02 22:33:00.807692 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.807644 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" Nov 02 22:33:00.807692 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.807674 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" event={"ID":"fd88af66-9693-4cd1-abd5-d2af89caa020","Type":"ContainerDied","Data":"c0bf40f63c8a02aced0aa4292f4d2d059e671c5df724c928cef3a0eea59d964c"} Nov 02 22:33:00.808012 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.807758 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2" event={"ID":"fd88af66-9693-4cd1-abd5-d2af89caa020","Type":"ContainerDied","Data":"34e3df37edb224135b28feec8ea8ed718923aa8fb6e54478a1d49ff570b8c25b"} Nov 02 22:33:00.808012 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.807781 5871 scope.go:122] "RemoveContainer" containerID="c0bf40f63c8a02aced0aa4292f4d2d059e671c5df724c928cef3a0eea59d964c" Nov 02 22:33:00.819500 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.819478 5871 scope.go:122] "RemoveContainer" containerID="c0bf40f63c8a02aced0aa4292f4d2d059e671c5df724c928cef3a0eea59d964c" Nov 02 22:33:00.819845 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:00.819826 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0bf40f63c8a02aced0aa4292f4d2d059e671c5df724c928cef3a0eea59d964c\": container with ID starting with c0bf40f63c8a02aced0aa4292f4d2d059e671c5df724c928cef3a0eea59d964c not found: ID does not exist" containerID="c0bf40f63c8a02aced0aa4292f4d2d059e671c5df724c928cef3a0eea59d964c" Nov 02 22:33:00.819909 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.819858 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0bf40f63c8a02aced0aa4292f4d2d059e671c5df724c928cef3a0eea59d964c"} err="failed to get container status \"c0bf40f63c8a02aced0aa4292f4d2d059e671c5df724c928cef3a0eea59d964c\": rpc error: code = NotFound desc = could not find container \"c0bf40f63c8a02aced0aa4292f4d2d059e671c5df724c928cef3a0eea59d964c\": container with ID starting with c0bf40f63c8a02aced0aa4292f4d2d059e671c5df724c928cef3a0eea59d964c not found: ID does not exist" Nov 02 22:33:00.977362 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:00.977318 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2"] Nov 02 22:33:01.025686 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:01.025643 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-ca85db-6ddbf7d965-g4hq2"] Nov 02 22:33:01.314626 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:01.314590 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd88af66-9693-4cd1-abd5-d2af89caa020" path="/var/lib/kubelet/pods/fd88af66-9693-4cd1-abd5-d2af89caa020/volumes" Nov 02 22:33:01.661480 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:01.661443 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:01.904187 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:01.904135 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" containerID="cri-o://2ee1e0497a1aa715e91b680a483f363e255ec8d54f2a2de15c8b3a5430a60452" gracePeriod=45 Nov 02 22:33:01.992220 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:01.992182 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:02.180741 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.180707 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:02.195539 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.195509 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:33:02.300662 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.300625 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-tmp-dir\") pod \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " Nov 02 22:33:02.300846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.300676 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b70d1226-ba5d-4cf0-af76-4852ce82bd76-tls-certs\" (UniqueName: \"kubernetes.io/secret/b70d1226-ba5d-4cf0-af76-4852ce82bd76-tls-certs\") pod \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " Nov 02 22:33:02.300846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.300834 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-model-cache\") pod \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " Nov 02 22:33:02.301036 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.300949 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-home\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-home\") pod \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " Nov 02 22:33:02.301036 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.301014 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-tmp-dir" pod "b70d1226-ba5d-4cf0-af76-4852ce82bd76" (UID: "b70d1226-ba5d-4cf0-af76-4852ce82bd76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:02.301143 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.301059 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-dshm\") pod \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " Nov 02 22:33:02.301143 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.301117 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70d1226-ba5d-4cf0-af76-4852ce82bd76-kube-api-access-zf7kk\" (UniqueName: \"kubernetes.io/projected/b70d1226-ba5d-4cf0-af76-4852ce82bd76-kube-api-access-zf7kk\") pod \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\" (UID: \"b70d1226-ba5d-4cf0-af76-4852ce82bd76\") " Nov 02 22:33:02.301250 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.301116 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-model-cache" pod "b70d1226-ba5d-4cf0-af76-4852ce82bd76" (UID: "b70d1226-ba5d-4cf0-af76-4852ce82bd76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:02.301250 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.301180 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-home" pod "b70d1226-ba5d-4cf0-af76-4852ce82bd76" (UID: "b70d1226-ba5d-4cf0-af76-4852ce82bd76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:02.301648 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.301626 5871 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-model-cache\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:02.301747 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.301651 5871 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-home\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:02.301747 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.301665 5871 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-tmp-dir\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:02.303182 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.303160 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70d1226-ba5d-4cf0-af76-4852ce82bd76-kube-api-access-zf7kk" pod "b70d1226-ba5d-4cf0-af76-4852ce82bd76" (UID: "b70d1226-ba5d-4cf0-af76-4852ce82bd76"). InnerVolumeSpecName "kube-api-access-zf7kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:02.303283 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.303188 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-dshm" pod "b70d1226-ba5d-4cf0-af76-4852ce82bd76" (UID: "b70d1226-ba5d-4cf0-af76-4852ce82bd76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:02.303283 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.303191 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b70d1226-ba5d-4cf0-af76-4852ce82bd76-tls-certs" pod "b70d1226-ba5d-4cf0-af76-4852ce82bd76" (UID: "b70d1226-ba5d-4cf0-af76-4852ce82bd76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:02.403562 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.403534 5871 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b70d1226-ba5d-4cf0-af76-4852ce82bd76-dshm\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:02.403562 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.403560 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf7kk\" (UniqueName: \"kubernetes.io/projected/b70d1226-ba5d-4cf0-af76-4852ce82bd76-kube-api-access-zf7kk\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:02.403562 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.403574 5871 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b70d1226-ba5d-4cf0-af76-4852ce82bd76-tls-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:02.413067 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.413032 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:02.592837 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.592755 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:02.769601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.769562 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:02.825198 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.825165 5871 generic.go:356] "Generic (PLEG): container finished" podID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerID="2ee1e0497a1aa715e91b680a483f363e255ec8d54f2a2de15c8b3a5430a60452" exitCode=0 Nov 02 22:33:02.825447 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.825246 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" Nov 02 22:33:02.825447 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.825260 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" event={"ID":"b70d1226-ba5d-4cf0-af76-4852ce82bd76","Type":"ContainerDied","Data":"2ee1e0497a1aa715e91b680a483f363e255ec8d54f2a2de15c8b3a5430a60452"} Nov 02 22:33:02.825447 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.825312 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k" event={"ID":"b70d1226-ba5d-4cf0-af76-4852ce82bd76","Type":"ContainerDied","Data":"ec7a9d9adaebd58cc002efca6afc6d12c3612f331bfef3948869f1a0e18604f1"} Nov 02 22:33:02.825447 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.825338 5871 scope.go:122] "RemoveContainer" containerID="2ee1e0497a1aa715e91b680a483f363e255ec8d54f2a2de15c8b3a5430a60452" Nov 02 22:33:02.836562 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.836537 5871 scope.go:122] "RemoveContainer" containerID="2ee1e0497a1aa715e91b680a483f363e255ec8d54f2a2de15c8b3a5430a60452" Nov 02 22:33:02.836921 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:02.836900 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee1e0497a1aa715e91b680a483f363e255ec8d54f2a2de15c8b3a5430a60452\": container with ID starting with 2ee1e0497a1aa715e91b680a483f363e255ec8d54f2a2de15c8b3a5430a60452 not found: ID does not exist" containerID="2ee1e0497a1aa715e91b680a483f363e255ec8d54f2a2de15c8b3a5430a60452" Nov 02 22:33:02.836988 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:02.836931 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee1e0497a1aa715e91b680a483f363e255ec8d54f2a2de15c8b3a5430a60452"} err="failed to get container status \"2ee1e0497a1aa715e91b680a483f363e255ec8d54f2a2de15c8b3a5430a60452\": rpc error: code = NotFound desc = could not find container \"2ee1e0497a1aa715e91b680a483f363e255ec8d54f2a2de15c8b3a5430a60452\": container with ID starting with 2ee1e0497a1aa715e91b680a483f363e255ec8d54f2a2de15c8b3a5430a60452 not found: ID does not exist" Nov 02 22:33:03.025184 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:03.025150 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k"] Nov 02 22:33:03.074917 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:03.074867 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-fe810f/distinct-2-w5-fe810f-kserve-d47f8fd6c-7zv5k"] Nov 02 22:33:03.317466 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:03.317374 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" path="/var/lib/kubelet/pods/b70d1226-ba5d-4cf0-af76-4852ce82bd76/volumes" Nov 02 22:33:05.730445 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:05.730389 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" containerID="cri-o://42c41e097e7777554b7eaf25f45e1a05d94685c760e342684658c3cd0baccb5c" gracePeriod=45 Nov 02 22:33:05.738360 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:05.738324 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" containerID="cri-o://b3cb0148bfe60edc2197e0272a90c6cc546a8ddb5a7ad6ac7c80fbef5bf53cd1" gracePeriod=45 Nov 02 22:33:05.765343 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:05.765303 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" containerID="cri-o://795daadd62dd8ecc61b165629aa0c8d2ece2503d52dce29b6605f1ecb5c8510e" gracePeriod=45 Nov 02 22:33:05.850288 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:05.850255 5871 generic.go:356] "Generic (PLEG): container finished" podID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerID="42c41e097e7777554b7eaf25f45e1a05d94685c760e342684658c3cd0baccb5c" exitCode=0 Nov 02 22:33:05.850450 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:05.850339 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" event={"ID":"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede","Type":"ContainerDied","Data":"42c41e097e7777554b7eaf25f45e1a05d94685c760e342684658c3cd0baccb5c"} Nov 02 22:33:05.875704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:05.875668 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": dial tcp 10.134.0.67:8000: connect: connection refused" Nov 02 22:33:06.089371 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.089340 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:33:06.093155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.093124 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:33:06.097627 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.097603 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:33:06.260694 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.260659 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-model-cache\") pod \"36cf5632-ebba-49b5-ad2c-40f5004570bd\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " Nov 02 22:33:06.260969 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.260706 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-kube-api-access-w27w2\" (UniqueName: \"kubernetes.io/projected/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-kube-api-access-w27w2\") pod \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " Nov 02 22:33:06.260969 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.260728 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-dshm\") pod \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " Nov 02 22:33:06.260969 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.260763 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-tmp-dir\") pod \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " Nov 02 22:33:06.260969 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.260790 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-dshm\") pod \"36cf5632-ebba-49b5-ad2c-40f5004570bd\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " Nov 02 22:33:06.261200 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.261071 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-model-cache" pod "36cf5632-ebba-49b5-ad2c-40f5004570bd" (UID: "36cf5632-ebba-49b5-ad2c-40f5004570bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:06.261200 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.261080 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-tmp-dir" pod "7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" (UID: "7c57bc80-ff20-4fc4-a19d-5dce6dc11ede"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:06.261200 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.261148 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36cf5632-ebba-49b5-ad2c-40f5004570bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/36cf5632-ebba-49b5-ad2c-40f5004570bd-tls-certs\") pod \"36cf5632-ebba-49b5-ad2c-40f5004570bd\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " Nov 02 22:33:06.261357 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.261238 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-tmp-dir\") pod \"d705b15d-5840-41da-b5d8-aad314aeb33b\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " Nov 02 22:33:06.261414 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.261359 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-dshm\") pod \"d705b15d-5840-41da-b5d8-aad314aeb33b\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " Nov 02 22:33:06.261465 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.261415 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-home\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-home\") pod \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " Nov 02 22:33:06.261516 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.261494 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-tmp-dir" pod "d705b15d-5840-41da-b5d8-aad314aeb33b" (UID: "d705b15d-5840-41da-b5d8-aad314aeb33b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:06.262882 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.262828 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-home" pod "7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" (UID: "7c57bc80-ff20-4fc4-a19d-5dce6dc11ede"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:06.262882 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.262855 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d705b15d-5840-41da-b5d8-aad314aeb33b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d705b15d-5840-41da-b5d8-aad314aeb33b-tls-certs\") pod \"d705b15d-5840-41da-b5d8-aad314aeb33b\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " Nov 02 22:33:06.263078 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.262934 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d705b15d-5840-41da-b5d8-aad314aeb33b-kube-api-access-c28tf\" (UniqueName: \"kubernetes.io/projected/d705b15d-5840-41da-b5d8-aad314aeb33b-kube-api-access-c28tf\") pod \"d705b15d-5840-41da-b5d8-aad314aeb33b\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " Nov 02 22:33:06.263078 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.262995 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-tls-certs\") pod \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " Nov 02 22:33:06.263078 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.263022 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-home\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-home\") pod \"d705b15d-5840-41da-b5d8-aad314aeb33b\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " Nov 02 22:33:06.263078 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.263060 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-tmp-dir\") pod \"36cf5632-ebba-49b5-ad2c-40f5004570bd\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " Nov 02 22:33:06.263290 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.263098 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-model-cache\") pod \"d705b15d-5840-41da-b5d8-aad314aeb33b\" (UID: \"d705b15d-5840-41da-b5d8-aad314aeb33b\") " Nov 02 22:33:06.263486 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.263138 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36cf5632-ebba-49b5-ad2c-40f5004570bd-kube-api-access-mbpkc\" (UniqueName: \"kubernetes.io/projected/36cf5632-ebba-49b5-ad2c-40f5004570bd-kube-api-access-mbpkc\") pod \"36cf5632-ebba-49b5-ad2c-40f5004570bd\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " Nov 02 22:33:06.263610 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.263551 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-home\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-home\") pod \"36cf5632-ebba-49b5-ad2c-40f5004570bd\" (UID: \"36cf5632-ebba-49b5-ad2c-40f5004570bd\") " Nov 02 22:33:06.263674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.263645 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-model-cache\") pod \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\" (UID: \"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede\") " Nov 02 22:33:06.263726 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.263687 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-home" pod "d705b15d-5840-41da-b5d8-aad314aeb33b" (UID: "d705b15d-5840-41da-b5d8-aad314aeb33b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:06.264850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.264049 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-model-cache" pod "d705b15d-5840-41da-b5d8-aad314aeb33b" (UID: "d705b15d-5840-41da-b5d8-aad314aeb33b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:06.264850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.264166 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-tmp-dir" pod "36cf5632-ebba-49b5-ad2c-40f5004570bd" (UID: "36cf5632-ebba-49b5-ad2c-40f5004570bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:06.264850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.264349 5871 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-home\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.264850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.264367 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-dshm" pod "d705b15d-5840-41da-b5d8-aad314aeb33b" (UID: "d705b15d-5840-41da-b5d8-aad314aeb33b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:06.264850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.264385 5871 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-home\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.264850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.264388 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36cf5632-ebba-49b5-ad2c-40f5004570bd-tls-certs" pod "36cf5632-ebba-49b5-ad2c-40f5004570bd" (UID: "36cf5632-ebba-49b5-ad2c-40f5004570bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:06.264850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.264400 5871 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-tmp-dir\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.264850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.264414 5871 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-model-cache\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.264850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.264417 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-dshm" pod "7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" (UID: "7c57bc80-ff20-4fc4-a19d-5dce6dc11ede"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:06.264850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.264423 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-kube-api-access-w27w2" pod "7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" (UID: "7c57bc80-ff20-4fc4-a19d-5dce6dc11ede"). InnerVolumeSpecName "kube-api-access-w27w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:06.264850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.264430 5871 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-model-cache\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.264850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.264435 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-model-cache" pod "7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" (UID: "7c57bc80-ff20-4fc4-a19d-5dce6dc11ede"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:06.264850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.264455 5871 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-tmp-dir\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.264850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.264466 5871 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-tmp-dir\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.264850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.264818 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-home" pod "36cf5632-ebba-49b5-ad2c-40f5004570bd" (UID: "36cf5632-ebba-49b5-ad2c-40f5004570bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:06.265658 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.265633 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-tls-certs" pod "7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" (UID: "7c57bc80-ff20-4fc4-a19d-5dce6dc11ede"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:06.265772 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.265638 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-dshm" pod "36cf5632-ebba-49b5-ad2c-40f5004570bd" (UID: "36cf5632-ebba-49b5-ad2c-40f5004570bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:06.266582 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.266556 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36cf5632-ebba-49b5-ad2c-40f5004570bd-kube-api-access-mbpkc" pod "36cf5632-ebba-49b5-ad2c-40f5004570bd" (UID: "36cf5632-ebba-49b5-ad2c-40f5004570bd"). InnerVolumeSpecName "kube-api-access-mbpkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:06.266847 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.266830 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d705b15d-5840-41da-b5d8-aad314aeb33b-tls-certs" pod "d705b15d-5840-41da-b5d8-aad314aeb33b" (UID: "d705b15d-5840-41da-b5d8-aad314aeb33b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:06.268113 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.268091 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d705b15d-5840-41da-b5d8-aad314aeb33b-kube-api-access-c28tf" pod "d705b15d-5840-41da-b5d8-aad314aeb33b" (UID: "d705b15d-5840-41da-b5d8-aad314aeb33b"). InnerVolumeSpecName "kube-api-access-c28tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:06.366604 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.366563 5871 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d705b15d-5840-41da-b5d8-aad314aeb33b-tls-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.366604 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.366607 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c28tf\" (UniqueName: \"kubernetes.io/projected/d705b15d-5840-41da-b5d8-aad314aeb33b-kube-api-access-c28tf\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.366806 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.366623 5871 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-tls-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.366806 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.366640 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbpkc\" (UniqueName: \"kubernetes.io/projected/36cf5632-ebba-49b5-ad2c-40f5004570bd-kube-api-access-mbpkc\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.366806 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.366654 5871 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-home\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.366806 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.366669 5871 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-model-cache\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.366806 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.366683 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w27w2\" (UniqueName: \"kubernetes.io/projected/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-kube-api-access-w27w2\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.366806 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.366696 5871 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede-dshm\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.366806 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.366709 5871 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36cf5632-ebba-49b5-ad2c-40f5004570bd-dshm\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.366806 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.366721 5871 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36cf5632-ebba-49b5-ad2c-40f5004570bd-tls-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.366806 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.366733 5871 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d705b15d-5840-41da-b5d8-aad314aeb33b-dshm\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:06.859223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.859185 5871 generic.go:356] "Generic (PLEG): container finished" podID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerID="795daadd62dd8ecc61b165629aa0c8d2ece2503d52dce29b6605f1ecb5c8510e" exitCode=0 Nov 02 22:33:06.859748 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.859230 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" Nov 02 22:33:06.859748 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.859274 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" event={"ID":"d705b15d-5840-41da-b5d8-aad314aeb33b","Type":"ContainerDied","Data":"795daadd62dd8ecc61b165629aa0c8d2ece2503d52dce29b6605f1ecb5c8510e"} Nov 02 22:33:06.859748 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.859325 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5" event={"ID":"d705b15d-5840-41da-b5d8-aad314aeb33b","Type":"ContainerDied","Data":"199e6a737a31ab3f598396bbed39e5f6d035378cf86406c4236e122271f8670a"} Nov 02 22:33:06.859748 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.859346 5871 scope.go:122] "RemoveContainer" containerID="795daadd62dd8ecc61b165629aa0c8d2ece2503d52dce29b6605f1ecb5c8510e" Nov 02 22:33:06.860945 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.860916 5871 generic.go:356] "Generic (PLEG): container finished" podID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerID="b3cb0148bfe60edc2197e0272a90c6cc546a8ddb5a7ad6ac7c80fbef5bf53cd1" exitCode=0 Nov 02 22:33:06.861072 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.860949 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" Nov 02 22:33:06.861072 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.860997 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" event={"ID":"36cf5632-ebba-49b5-ad2c-40f5004570bd","Type":"ContainerDied","Data":"b3cb0148bfe60edc2197e0272a90c6cc546a8ddb5a7ad6ac7c80fbef5bf53cd1"} Nov 02 22:33:06.861072 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.861039 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb" event={"ID":"36cf5632-ebba-49b5-ad2c-40f5004570bd","Type":"ContainerDied","Data":"29be60302f21def038fedf9c1347ce7f7fad555191f12f1883f4ac734a6df0d2"} Nov 02 22:33:06.862846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.862827 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" Nov 02 22:33:06.862959 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.862840 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq" event={"ID":"7c57bc80-ff20-4fc4-a19d-5dce6dc11ede","Type":"ContainerDied","Data":"1284a6439dfb4dbc834fd9328e6cd58d91b9924f54fa0474a2bbaa3e117b1c12"} Nov 02 22:33:06.871117 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.871087 5871 scope.go:122] "RemoveContainer" containerID="795daadd62dd8ecc61b165629aa0c8d2ece2503d52dce29b6605f1ecb5c8510e" Nov 02 22:33:06.871539 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:06.871453 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795daadd62dd8ecc61b165629aa0c8d2ece2503d52dce29b6605f1ecb5c8510e\": container with ID starting with 795daadd62dd8ecc61b165629aa0c8d2ece2503d52dce29b6605f1ecb5c8510e not found: ID does not exist" containerID="795daadd62dd8ecc61b165629aa0c8d2ece2503d52dce29b6605f1ecb5c8510e" Nov 02 22:33:06.871539 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.871496 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795daadd62dd8ecc61b165629aa0c8d2ece2503d52dce29b6605f1ecb5c8510e"} err="failed to get container status \"795daadd62dd8ecc61b165629aa0c8d2ece2503d52dce29b6605f1ecb5c8510e\": rpc error: code = NotFound desc = could not find container \"795daadd62dd8ecc61b165629aa0c8d2ece2503d52dce29b6605f1ecb5c8510e\": container with ID starting with 795daadd62dd8ecc61b165629aa0c8d2ece2503d52dce29b6605f1ecb5c8510e not found: ID does not exist" Nov 02 22:33:06.871539 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.871522 5871 scope.go:122] "RemoveContainer" containerID="b3cb0148bfe60edc2197e0272a90c6cc546a8ddb5a7ad6ac7c80fbef5bf53cd1" Nov 02 22:33:06.883317 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.883295 5871 scope.go:122] "RemoveContainer" containerID="b3cb0148bfe60edc2197e0272a90c6cc546a8ddb5a7ad6ac7c80fbef5bf53cd1" Nov 02 22:33:06.883651 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:06.883630 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3cb0148bfe60edc2197e0272a90c6cc546a8ddb5a7ad6ac7c80fbef5bf53cd1\": container with ID starting with b3cb0148bfe60edc2197e0272a90c6cc546a8ddb5a7ad6ac7c80fbef5bf53cd1 not found: ID does not exist" containerID="b3cb0148bfe60edc2197e0272a90c6cc546a8ddb5a7ad6ac7c80fbef5bf53cd1" Nov 02 22:33:06.883705 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.883662 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3cb0148bfe60edc2197e0272a90c6cc546a8ddb5a7ad6ac7c80fbef5bf53cd1"} err="failed to get container status \"b3cb0148bfe60edc2197e0272a90c6cc546a8ddb5a7ad6ac7c80fbef5bf53cd1\": rpc error: code = NotFound desc = could not find container \"b3cb0148bfe60edc2197e0272a90c6cc546a8ddb5a7ad6ac7c80fbef5bf53cd1\": container with ID starting with b3cb0148bfe60edc2197e0272a90c6cc546a8ddb5a7ad6ac7c80fbef5bf53cd1 not found: ID does not exist" Nov 02 22:33:06.883705 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.883680 5871 scope.go:122] "RemoveContainer" containerID="42c41e097e7777554b7eaf25f45e1a05d94685c760e342684658c3cd0baccb5c" Nov 02 22:33:06.895144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:06.895113 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:07.027771 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:07.027687 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq"] Nov 02 22:33:07.031513 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:07.031487 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/distinct-2-w3-98a222-kserve-7dfc57c499-6d9hq"] Nov 02 22:33:07.139962 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:07.139927 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb"] Nov 02 22:33:07.145265 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:07.144607 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/distinct-w3-98a222-kserve-65475c68cb-fsltb"] Nov 02 22:33:07.189510 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:07.189474 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:07.256129 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:07.256097 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5"] Nov 02 22:33:07.260925 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:07.260848 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-98a222/simulator-w3-98a222-kserve-749cbbb68-zpfs5"] Nov 02 22:33:07.313546 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:07.313468 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" path="/var/lib/kubelet/pods/36cf5632-ebba-49b5-ad2c-40f5004570bd/volumes" Nov 02 22:33:07.313987 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:07.313972 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" path="/var/lib/kubelet/pods/7c57bc80-ff20-4fc4-a19d-5dce6dc11ede/volumes" Nov 02 22:33:07.314387 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:07.314374 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" path="/var/lib/kubelet/pods/d705b15d-5840-41da-b5d8-aad314aeb33b/volumes" Nov 02 22:33:08.793802 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:08.793767 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:08.981319 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:08.981283 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:09.137095 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:09.137063 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5"] Nov 02 22:33:09.137591 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:09.137556 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" podUID="3782bc26-cb56-41c1-b1d2-608d185145fc" containerName="istio-proxy" containerID="cri-o://62a4239770c3f7918be6f2894c12468c36ea33ea38c646c8e31fe68a6fab1eb0" gracePeriod=30 Nov 02 22:33:09.157305 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:09.157267 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz"] Nov 02 22:33:09.157495 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:09.157473 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" podUID="4824cb6c-15d6-4c48-8824-d7bf7b7932f1" containerName="istio-proxy" containerID="cri-o://6b3a2646fdbe2076bb588f5a9ce673950c8be725caea1fee22d5dc013b224c28" gracePeriod=30 Nov 02 22:33:09.169351 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:09.169322 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:09.178445 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:09.178414 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9"] Nov 02 22:33:09.178644 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:09.178625 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" podUID="3aa92502-a30d-4230-9338-80d73e06c66f" containerName="istio-proxy" containerID="cri-o://6c9d84c786093b37fc77810c778bf3587e7bf59710b414099c9faeaca1f6bc99" gracePeriod=30 Nov 02 22:33:12.290279 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:12.290243 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:13.771239 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:13.771189 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:33:14.462752 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.462119 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:33:14.523975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.523947 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:33:14.527444 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.527418 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:33:14.586832 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.586800 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-podinfo\") pod \"3782bc26-cb56-41c1-b1d2-608d185145fc\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " Nov 02 22:33:14.587059 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.586846 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-credential-socket\") pod \"3aa92502-a30d-4230-9338-80d73e06c66f\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " Nov 02 22:33:14.587059 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.586868 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-credential-socket\") pod \"3782bc26-cb56-41c1-b1d2-608d185145fc\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " Nov 02 22:33:14.587059 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.586922 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-kube-api-access-wjk2h\" (UniqueName: \"kubernetes.io/projected/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-kube-api-access-wjk2h\") pod \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " Nov 02 22:33:14.587059 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.586961 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3aa92502-a30d-4230-9338-80d73e06c66f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa92502-a30d-4230-9338-80d73e06c66f-istio-podinfo\") pod \"3aa92502-a30d-4230-9338-80d73e06c66f\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " Nov 02 22:33:14.587296 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587132 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-istio-data\") pod \"3aa92502-a30d-4230-9338-80d73e06c66f\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " Nov 02 22:33:14.587296 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587178 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-workload-socket\") pod \"3782bc26-cb56-41c1-b1d2-608d185145fc\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " Nov 02 22:33:14.587296 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587203 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-credential-socket" pod "3aa92502-a30d-4230-9338-80d73e06c66f" (UID: "3aa92502-a30d-4230-9338-80d73e06c66f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.587296 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587225 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-credential-socket" pod "3782bc26-cb56-41c1-b1d2-608d185145fc" (UID: "3782bc26-cb56-41c1-b1d2-608d185145fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.587296 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587233 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-envoy\") pod \"3782bc26-cb56-41c1-b1d2-608d185145fc\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " Nov 02 22:33:14.587566 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587297 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-token\" (UniqueName: \"kubernetes.io/projected/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-token\") pod \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " Nov 02 22:33:14.587566 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587378 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa92502-a30d-4230-9338-80d73e06c66f-istio-token\" (UniqueName: \"kubernetes.io/projected/3aa92502-a30d-4230-9338-80d73e06c66f-istio-token\") pod \"3aa92502-a30d-4230-9338-80d73e06c66f\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " Nov 02 22:33:14.587566 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587399 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-istio-data" pod "3aa92502-a30d-4230-9338-80d73e06c66f" (UID: "3aa92502-a30d-4230-9338-80d73e06c66f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.587566 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587430 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-data\") pod \"3782bc26-cb56-41c1-b1d2-608d185145fc\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " Nov 02 22:33:14.587566 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587479 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3782bc26-cb56-41c1-b1d2-608d185145fc-kube-api-access-5jtvd\" (UniqueName: \"kubernetes.io/projected/3782bc26-cb56-41c1-b1d2-608d185145fc-kube-api-access-5jtvd\") pod \"3782bc26-cb56-41c1-b1d2-608d185145fc\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " Nov 02 22:33:14.587566 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587517 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-workload-certs\") pod \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " Nov 02 22:33:14.587566 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587550 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-credential-socket\") pod \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " Nov 02 22:33:14.587984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587623 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa92502-a30d-4230-9338-80d73e06c66f-kube-api-access-72qg6\" (UniqueName: \"kubernetes.io/projected/3aa92502-a30d-4230-9338-80d73e06c66f-kube-api-access-72qg6\") pod \"3aa92502-a30d-4230-9338-80d73e06c66f\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " Nov 02 22:33:14.587984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587667 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-data\") pod \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " Nov 02 22:33:14.587984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587698 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3782bc26-cb56-41c1-b1d2-608d185145fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3782bc26-cb56-41c1-b1d2-608d185145fc-istiod-ca-cert\") pod \"3782bc26-cb56-41c1-b1d2-608d185145fc\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " Nov 02 22:33:14.587984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587728 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-token\" (UniqueName: \"kubernetes.io/projected/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-token\") pod \"3782bc26-cb56-41c1-b1d2-608d185145fc\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " Nov 02 22:33:14.587984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587800 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa92502-a30d-4230-9338-80d73e06c66f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa92502-a30d-4230-9338-80d73e06c66f-istiod-ca-cert\") pod \"3aa92502-a30d-4230-9338-80d73e06c66f\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " Nov 02 22:33:14.587984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587907 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-istio-envoy\") pod \"3aa92502-a30d-4230-9338-80d73e06c66f\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " Nov 02 22:33:14.587984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587951 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-workload-certs\") pod \"3782bc26-cb56-41c1-b1d2-608d185145fc\" (UID: \"3782bc26-cb56-41c1-b1d2-608d185145fc\") " Nov 02 22:33:14.587984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.587986 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-envoy\") pod \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " Nov 02 22:33:14.588419 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.588058 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-podinfo\") pod \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " Nov 02 22:33:14.588419 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.588091 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-workload-socket\") pod \"3aa92502-a30d-4230-9338-80d73e06c66f\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " Nov 02 22:33:14.588419 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.588127 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-workload-socket\") pod \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " Nov 02 22:33:14.588419 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.588254 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-workload-certs\") pod \"3aa92502-a30d-4230-9338-80d73e06c66f\" (UID: \"3aa92502-a30d-4230-9338-80d73e06c66f\") " Nov 02 22:33:14.588419 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.588301 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istiod-ca-cert\") pod \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\" (UID: \"4824cb6c-15d6-4c48-8824-d7bf7b7932f1\") " Nov 02 22:33:14.589065 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.589038 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.589188 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.589069 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.589188 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.589086 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.589461 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.589440 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istiod-ca-cert" pod "4824cb6c-15d6-4c48-8824-d7bf7b7932f1" (UID: "4824cb6c-15d6-4c48-8824-d7bf7b7932f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:14.590392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.589626 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-podinfo" pod "3782bc26-cb56-41c1-b1d2-608d185145fc" (UID: "3782bc26-cb56-41c1-b1d2-608d185145fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:14.590392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.589698 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-data" pod "4824cb6c-15d6-4c48-8824-d7bf7b7932f1" (UID: "4824cb6c-15d6-4c48-8824-d7bf7b7932f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.590392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.589943 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-envoy" pod "3782bc26-cb56-41c1-b1d2-608d185145fc" (UID: "3782bc26-cb56-41c1-b1d2-608d185145fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.590392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.589942 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3aa92502-a30d-4230-9338-80d73e06c66f-istio-podinfo" pod "3aa92502-a30d-4230-9338-80d73e06c66f" (UID: "3aa92502-a30d-4230-9338-80d73e06c66f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:14.590392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.590048 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-token" pod "4824cb6c-15d6-4c48-8824-d7bf7b7932f1" (UID: "4824cb6c-15d6-4c48-8824-d7bf7b7932f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:14.590392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.590059 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3782bc26-cb56-41c1-b1d2-608d185145fc-istiod-ca-cert" pod "3782bc26-cb56-41c1-b1d2-608d185145fc" (UID: "3782bc26-cb56-41c1-b1d2-608d185145fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:14.590392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.590199 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-workload-socket" pod "3782bc26-cb56-41c1-b1d2-608d185145fc" (UID: "3782bc26-cb56-41c1-b1d2-608d185145fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.590392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.590387 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa92502-a30d-4230-9338-80d73e06c66f-kube-api-access-72qg6" pod "3aa92502-a30d-4230-9338-80d73e06c66f" (UID: "3aa92502-a30d-4230-9338-80d73e06c66f"). InnerVolumeSpecName "kube-api-access-72qg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:14.590933 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.590598 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-workload-certs" pod "4824cb6c-15d6-4c48-8824-d7bf7b7932f1" (UID: "4824cb6c-15d6-4c48-8824-d7bf7b7932f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.590933 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.590611 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa92502-a30d-4230-9338-80d73e06c66f-istiod-ca-cert" pod "3aa92502-a30d-4230-9338-80d73e06c66f" (UID: "3aa92502-a30d-4230-9338-80d73e06c66f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:14.590933 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.590744 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-kube-api-access-wjk2h" pod "4824cb6c-15d6-4c48-8824-d7bf7b7932f1" (UID: "4824cb6c-15d6-4c48-8824-d7bf7b7932f1"). InnerVolumeSpecName "kube-api-access-wjk2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:14.590933 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.590766 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-credential-socket" pod "4824cb6c-15d6-4c48-8824-d7bf7b7932f1" (UID: "4824cb6c-15d6-4c48-8824-d7bf7b7932f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.591175 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.590996 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-data" pod "3782bc26-cb56-41c1-b1d2-608d185145fc" (UID: "3782bc26-cb56-41c1-b1d2-608d185145fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.591175 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.590998 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-workload-socket" pod "4824cb6c-15d6-4c48-8824-d7bf7b7932f1" (UID: "4824cb6c-15d6-4c48-8824-d7bf7b7932f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.591293 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.591175 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-workload-socket" pod "3aa92502-a30d-4230-9338-80d73e06c66f" (UID: "3aa92502-a30d-4230-9338-80d73e06c66f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.591293 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.591189 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-workload-certs" pod "3aa92502-a30d-4230-9338-80d73e06c66f" (UID: "3aa92502-a30d-4230-9338-80d73e06c66f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.591414 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.591364 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-workload-certs" pod "3782bc26-cb56-41c1-b1d2-608d185145fc" (UID: "3782bc26-cb56-41c1-b1d2-608d185145fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.591961 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.591936 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3782bc26-cb56-41c1-b1d2-608d185145fc-kube-api-access-5jtvd" pod "3782bc26-cb56-41c1-b1d2-608d185145fc" (UID: "3782bc26-cb56-41c1-b1d2-608d185145fc"). InnerVolumeSpecName "kube-api-access-5jtvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:14.592313 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.592294 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-envoy" pod "4824cb6c-15d6-4c48-8824-d7bf7b7932f1" (UID: "4824cb6c-15d6-4c48-8824-d7bf7b7932f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.592631 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.592613 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-podinfo" pod "4824cb6c-15d6-4c48-8824-d7bf7b7932f1" (UID: "4824cb6c-15d6-4c48-8824-d7bf7b7932f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:14.592799 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.592779 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-token" pod "3782bc26-cb56-41c1-b1d2-608d185145fc" (UID: "3782bc26-cb56-41c1-b1d2-608d185145fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:14.593405 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.593385 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa92502-a30d-4230-9338-80d73e06c66f-istio-token" pod "3aa92502-a30d-4230-9338-80d73e06c66f" (UID: "3aa92502-a30d-4230-9338-80d73e06c66f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:14.593485 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.593450 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-istio-envoy" pod "3aa92502-a30d-4230-9338-80d73e06c66f" (UID: "3aa92502-a30d-4230-9338-80d73e06c66f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:14.690867 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.690770 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.690867 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.690803 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa92502-a30d-4230-9338-80d73e06c66f-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.690867 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.690819 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.690867 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.690832 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jtvd\" (UniqueName: \"kubernetes.io/projected/3782bc26-cb56-41c1-b1d2-608d185145fc-kube-api-access-5jtvd\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.690867 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.690846 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.690867 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.690859 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.690885 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72qg6\" (UniqueName: \"kubernetes.io/projected/3aa92502-a30d-4230-9338-80d73e06c66f-kube-api-access-72qg6\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.690899 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.690910 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3782bc26-cb56-41c1-b1d2-608d185145fc-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.690922 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.690934 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa92502-a30d-4230-9338-80d73e06c66f-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.690948 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.690959 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.690972 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.690985 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.690997 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.691009 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.691022 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa92502-a30d-4230-9338-80d73e06c66f-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.691034 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.691047 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.691061 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjk2h\" (UniqueName: \"kubernetes.io/projected/4824cb6c-15d6-4c48-8824-d7bf7b7932f1-kube-api-access-wjk2h\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.691073 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa92502-a30d-4230-9338-80d73e06c66f-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.691087 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.691252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.691101 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3782bc26-cb56-41c1-b1d2-608d185145fc-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:14.926780 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.926738 5871 generic.go:356] "Generic (PLEG): container finished" podID="4824cb6c-15d6-4c48-8824-d7bf7b7932f1" containerID="6b3a2646fdbe2076bb588f5a9ce673950c8be725caea1fee22d5dc013b224c28" exitCode=0 Nov 02 22:33:14.927249 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.926791 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" Nov 02 22:33:14.927249 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.926822 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" event={"ID":"4824cb6c-15d6-4c48-8824-d7bf7b7932f1","Type":"ContainerDied","Data":"6b3a2646fdbe2076bb588f5a9ce673950c8be725caea1fee22d5dc013b224c28"} Nov 02 22:33:14.927249 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.926897 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz" event={"ID":"4824cb6c-15d6-4c48-8824-d7bf7b7932f1","Type":"ContainerDied","Data":"abbb1f984cd20e5f7f1fa0d6f92df4a98744d9a3d5e61646888c264561d21a58"} Nov 02 22:33:14.927249 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.926925 5871 scope.go:122] "RemoveContainer" containerID="6b3a2646fdbe2076bb588f5a9ce673950c8be725caea1fee22d5dc013b224c28" Nov 02 22:33:14.928539 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.928507 5871 generic.go:356] "Generic (PLEG): container finished" podID="3aa92502-a30d-4230-9338-80d73e06c66f" containerID="6c9d84c786093b37fc77810c778bf3587e7bf59710b414099c9faeaca1f6bc99" exitCode=0 Nov 02 22:33:14.928645 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.928564 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" Nov 02 22:33:14.928703 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.928637 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" event={"ID":"3aa92502-a30d-4230-9338-80d73e06c66f","Type":"ContainerDied","Data":"6c9d84c786093b37fc77810c778bf3587e7bf59710b414099c9faeaca1f6bc99"} Nov 02 22:33:14.928703 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.928684 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9" event={"ID":"3aa92502-a30d-4230-9338-80d73e06c66f","Type":"ContainerDied","Data":"aaf5cb8e0b1d5f2ff9c3f3f6750e794044225192d8e2889488d8282bf1a12a2b"} Nov 02 22:33:14.930216 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.930193 5871 generic.go:356] "Generic (PLEG): container finished" podID="3782bc26-cb56-41c1-b1d2-608d185145fc" containerID="62a4239770c3f7918be6f2894c12468c36ea33ea38c646c8e31fe68a6fab1eb0" exitCode=0 Nov 02 22:33:14.930320 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.930230 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" event={"ID":"3782bc26-cb56-41c1-b1d2-608d185145fc","Type":"ContainerDied","Data":"62a4239770c3f7918be6f2894c12468c36ea33ea38c646c8e31fe68a6fab1eb0"} Nov 02 22:33:14.930320 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.930242 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" Nov 02 22:33:14.930320 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.930269 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5" event={"ID":"3782bc26-cb56-41c1-b1d2-608d185145fc","Type":"ContainerDied","Data":"80d1be7855a0f1ec4f7ffb6473a7c0ae8012b31fe355d6e30466fe65305480d6"} Nov 02 22:33:14.938054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.937957 5871 scope.go:122] "RemoveContainer" containerID="6b3a2646fdbe2076bb588f5a9ce673950c8be725caea1fee22d5dc013b224c28" Nov 02 22:33:14.938304 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:14.938276 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3a2646fdbe2076bb588f5a9ce673950c8be725caea1fee22d5dc013b224c28\": container with ID starting with 6b3a2646fdbe2076bb588f5a9ce673950c8be725caea1fee22d5dc013b224c28 not found: ID does not exist" containerID="6b3a2646fdbe2076bb588f5a9ce673950c8be725caea1fee22d5dc013b224c28" Nov 02 22:33:14.938395 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.938318 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3a2646fdbe2076bb588f5a9ce673950c8be725caea1fee22d5dc013b224c28"} err="failed to get container status \"6b3a2646fdbe2076bb588f5a9ce673950c8be725caea1fee22d5dc013b224c28\": rpc error: code = NotFound desc = could not find container \"6b3a2646fdbe2076bb588f5a9ce673950c8be725caea1fee22d5dc013b224c28\": container with ID starting with 6b3a2646fdbe2076bb588f5a9ce673950c8be725caea1fee22d5dc013b224c28 not found: ID does not exist" Nov 02 22:33:14.938395 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.938343 5871 scope.go:122] "RemoveContainer" containerID="6c9d84c786093b37fc77810c778bf3587e7bf59710b414099c9faeaca1f6bc99" Nov 02 22:33:14.949830 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.949810 5871 scope.go:122] "RemoveContainer" containerID="6c9d84c786093b37fc77810c778bf3587e7bf59710b414099c9faeaca1f6bc99" Nov 02 22:33:14.950205 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:14.950186 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9d84c786093b37fc77810c778bf3587e7bf59710b414099c9faeaca1f6bc99\": container with ID starting with 6c9d84c786093b37fc77810c778bf3587e7bf59710b414099c9faeaca1f6bc99 not found: ID does not exist" containerID="6c9d84c786093b37fc77810c778bf3587e7bf59710b414099c9faeaca1f6bc99" Nov 02 22:33:14.950287 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.950217 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9d84c786093b37fc77810c778bf3587e7bf59710b414099c9faeaca1f6bc99"} err="failed to get container status \"6c9d84c786093b37fc77810c778bf3587e7bf59710b414099c9faeaca1f6bc99\": rpc error: code = NotFound desc = could not find container \"6c9d84c786093b37fc77810c778bf3587e7bf59710b414099c9faeaca1f6bc99\": container with ID starting with 6c9d84c786093b37fc77810c778bf3587e7bf59710b414099c9faeaca1f6bc99 not found: ID does not exist" Nov 02 22:33:14.950287 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.950243 5871 scope.go:122] "RemoveContainer" containerID="62a4239770c3f7918be6f2894c12468c36ea33ea38c646c8e31fe68a6fab1eb0" Nov 02 22:33:14.960663 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.960635 5871 scope.go:122] "RemoveContainer" containerID="62a4239770c3f7918be6f2894c12468c36ea33ea38c646c8e31fe68a6fab1eb0" Nov 02 22:33:14.961020 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:14.960993 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a4239770c3f7918be6f2894c12468c36ea33ea38c646c8e31fe68a6fab1eb0\": container with ID starting with 62a4239770c3f7918be6f2894c12468c36ea33ea38c646c8e31fe68a6fab1eb0 not found: ID does not exist" containerID="62a4239770c3f7918be6f2894c12468c36ea33ea38c646c8e31fe68a6fab1eb0" Nov 02 22:33:14.961107 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:14.961031 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a4239770c3f7918be6f2894c12468c36ea33ea38c646c8e31fe68a6fab1eb0"} err="failed to get container status \"62a4239770c3f7918be6f2894c12468c36ea33ea38c646c8e31fe68a6fab1eb0\": rpc error: code = NotFound desc = could not find container \"62a4239770c3f7918be6f2894c12468c36ea33ea38c646c8e31fe68a6fab1eb0\": container with ID starting with 62a4239770c3f7918be6f2894c12468c36ea33ea38c646c8e31fe68a6fab1eb0 not found: ID does not exist" Nov 02 22:33:15.096160 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:15.096119 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz"] Nov 02 22:33:15.100621 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:15.100596 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-xz5gz"] Nov 02 22:33:15.214150 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:15.214066 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9"] Nov 02 22:33:15.217602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:15.217571 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-v2gw9"] Nov 02 22:33:15.314970 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:15.314939 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa92502-a30d-4230-9338-80d73e06c66f" path="/var/lib/kubelet/pods/3aa92502-a30d-4230-9338-80d73e06c66f/volumes" Nov 02 22:33:15.315448 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:15.315435 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4824cb6c-15d6-4c48-8824-d7bf7b7932f1" path="/var/lib/kubelet/pods/4824cb6c-15d6-4c48-8824-d7bf7b7932f1/volumes" Nov 02 22:33:15.329116 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:15.329084 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5"] Nov 02 22:33:15.333137 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:15.333110 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ca85db-openshift-default-6bdff5f4df-gdxl5"] Nov 02 22:33:16.555197 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:16.555159 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:17.318651 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:17.318607 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3782bc26-cb56-41c1-b1d2-608d185145fc" path="/var/lib/kubelet/pods/3782bc26-cb56-41c1-b1d2-608d185145fc/volumes" Nov 02 22:33:17.631901 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:17.631845 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4"] Nov 02 22:33:17.632350 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:17.632102 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" podUID="16b4519b-9df1-4c63-9f22-d46e804efef4" containerName="maas-api" containerID="cri-o://0bb1adef8fd6e98286a2f10cdfa65e1a5f9fcbbfed54021ae6c756987b8011a1" gracePeriod=30 Nov 02 22:33:17.765656 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:17.765624 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:17.945240 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:17.945213 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:33:17.958926 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:17.958889 5871 generic.go:356] "Generic (PLEG): container finished" podID="16b4519b-9df1-4c63-9f22-d46e804efef4" containerID="0bb1adef8fd6e98286a2f10cdfa65e1a5f9fcbbfed54021ae6c756987b8011a1" exitCode=0 Nov 02 22:33:17.959123 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:17.958962 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" Nov 02 22:33:17.959123 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:17.958983 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" event={"ID":"16b4519b-9df1-4c63-9f22-d46e804efef4","Type":"ContainerDied","Data":"0bb1adef8fd6e98286a2f10cdfa65e1a5f9fcbbfed54021ae6c756987b8011a1"} Nov 02 22:33:17.959123 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:17.959041 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4" event={"ID":"16b4519b-9df1-4c63-9f22-d46e804efef4","Type":"ContainerDied","Data":"2ce6b2e9a3dcc8953df976c09fbe8d6d3b2c19986f65b4b61b7cb63ff5669355"} Nov 02 22:33:17.959123 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:17.959075 5871 scope.go:122] "RemoveContainer" containerID="0bb1adef8fd6e98286a2f10cdfa65e1a5f9fcbbfed54021ae6c756987b8011a1" Nov 02 22:33:17.969278 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:17.969252 5871 scope.go:122] "RemoveContainer" containerID="0bb1adef8fd6e98286a2f10cdfa65e1a5f9fcbbfed54021ae6c756987b8011a1" Nov 02 22:33:17.969597 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:17.969571 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb1adef8fd6e98286a2f10cdfa65e1a5f9fcbbfed54021ae6c756987b8011a1\": container with ID starting with 0bb1adef8fd6e98286a2f10cdfa65e1a5f9fcbbfed54021ae6c756987b8011a1 not found: ID does not exist" containerID="0bb1adef8fd6e98286a2f10cdfa65e1a5f9fcbbfed54021ae6c756987b8011a1" Nov 02 22:33:17.969650 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:17.969609 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb1adef8fd6e98286a2f10cdfa65e1a5f9fcbbfed54021ae6c756987b8011a1"} err="failed to get container status \"0bb1adef8fd6e98286a2f10cdfa65e1a5f9fcbbfed54021ae6c756987b8011a1\": rpc error: code = NotFound desc = could not find container \"0bb1adef8fd6e98286a2f10cdfa65e1a5f9fcbbfed54021ae6c756987b8011a1\": container with ID starting with 0bb1adef8fd6e98286a2f10cdfa65e1a5f9fcbbfed54021ae6c756987b8011a1 not found: ID does not exist" Nov 02 22:33:18.049479 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.049444 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-metrics-certs\") pod \"16b4519b-9df1-4c63-9f22-d46e804efef4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " Nov 02 22:33:18.049737 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.049707 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b4519b-9df1-4c63-9f22-d46e804efef4-kube-api-access-4p2g5\" (UniqueName: \"kubernetes.io/projected/16b4519b-9df1-4c63-9f22-d46e804efef4-kube-api-access-4p2g5\") pod \"16b4519b-9df1-4c63-9f22-d46e804efef4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " Nov 02 22:33:18.049819 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.049780 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-maas-api-tls\") pod \"16b4519b-9df1-4c63-9f22-d46e804efef4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " Nov 02 22:33:18.049887 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.049830 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b4519b-9df1-4c63-9f22-d46e804efef4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b4519b-9df1-4c63-9f22-d46e804efef4-openshift-service-ca\") pod \"16b4519b-9df1-4c63-9f22-d46e804efef4\" (UID: \"16b4519b-9df1-4c63-9f22-d46e804efef4\") " Nov 02 22:33:18.050811 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.050776 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b4519b-9df1-4c63-9f22-d46e804efef4-openshift-service-ca" pod "16b4519b-9df1-4c63-9f22-d46e804efef4" (UID: "16b4519b-9df1-4c63-9f22-d46e804efef4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:18.069467 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.066201 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b4519b-9df1-4c63-9f22-d46e804efef4-kube-api-access-4p2g5" pod "16b4519b-9df1-4c63-9f22-d46e804efef4" (UID: "16b4519b-9df1-4c63-9f22-d46e804efef4"). InnerVolumeSpecName "kube-api-access-4p2g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:18.069467 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.066305 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-metrics-certs" pod "16b4519b-9df1-4c63-9f22-d46e804efef4" (UID: "16b4519b-9df1-4c63-9f22-d46e804efef4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:18.071054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.071016 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-maas-api-tls" pod "16b4519b-9df1-4c63-9f22-d46e804efef4" (UID: "16b4519b-9df1-4c63-9f22-d46e804efef4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:18.131975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.131931 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:18.152333 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.152244 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p2g5\" (UniqueName: \"kubernetes.io/projected/16b4519b-9df1-4c63-9f22-d46e804efef4-kube-api-access-4p2g5\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:18.152333 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.152283 5871 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-maas-api-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:18.152333 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.152301 5871 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16b4519b-9df1-4c63-9f22-d46e804efef4-openshift-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:18.152333 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.152314 5871 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16b4519b-9df1-4c63-9f22-d46e804efef4-metrics-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:18.384258 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.384226 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4"] Nov 02 22:33:18.387378 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.387345 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-98a222-7b46f76c56-6qfr4"] Nov 02 22:33:18.436824 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.436723 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:18.851155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.851052 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7"] Nov 02 22:33:18.851587 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:18.851263 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" podUID="b03723b8-b2e9-4f09-9054-c069f1f5f45c" containerName="maas-api" containerID="cri-o://3502d501c38037f70e8dfda6944e3aad15054be13451225808c698871a7b146b" gracePeriod=30 Nov 02 22:33:19.022369 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.022331 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:19.186891 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.183621 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:33:19.202064 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.202026 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:19.347073 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.344814 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b4519b-9df1-4c63-9f22-d46e804efef4" path="/var/lib/kubelet/pods/16b4519b-9df1-4c63-9f22-d46e804efef4/volumes" Nov 02 22:33:19.370965 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.370932 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-maas-api-tls\") pod \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " Nov 02 22:33:19.371150 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.371029 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-metrics-certs\") pod \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " Nov 02 22:33:19.371150 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.371070 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b03723b8-b2e9-4f09-9054-c069f1f5f45c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03723b8-b2e9-4f09-9054-c069f1f5f45c-openshift-service-ca\") pod \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " Nov 02 22:33:19.371150 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.371145 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b03723b8-b2e9-4f09-9054-c069f1f5f45c-kube-api-access-dwgwc\" (UniqueName: \"kubernetes.io/projected/b03723b8-b2e9-4f09-9054-c069f1f5f45c-kube-api-access-dwgwc\") pod \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\" (UID: \"b03723b8-b2e9-4f09-9054-c069f1f5f45c\") " Nov 02 22:33:19.371572 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.371544 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03723b8-b2e9-4f09-9054-c069f1f5f45c-openshift-service-ca" pod "b03723b8-b2e9-4f09-9054-c069f1f5f45c" (UID: "b03723b8-b2e9-4f09-9054-c069f1f5f45c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:19.371832 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.371809 5871 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03723b8-b2e9-4f09-9054-c069f1f5f45c-openshift-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:19.373508 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.373481 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-metrics-certs" pod "b03723b8-b2e9-4f09-9054-c069f1f5f45c" (UID: "b03723b8-b2e9-4f09-9054-c069f1f5f45c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:19.373597 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.373523 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-maas-api-tls" pod "b03723b8-b2e9-4f09-9054-c069f1f5f45c" (UID: "b03723b8-b2e9-4f09-9054-c069f1f5f45c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:19.373597 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.373543 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03723b8-b2e9-4f09-9054-c069f1f5f45c-kube-api-access-dwgwc" pod "b03723b8-b2e9-4f09-9054-c069f1f5f45c" (UID: "b03723b8-b2e9-4f09-9054-c069f1f5f45c"). InnerVolumeSpecName "kube-api-access-dwgwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:19.379992 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.379966 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:19.473252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.473197 5871 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-maas-api-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:19.473252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.473252 5871 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b03723b8-b2e9-4f09-9054-c069f1f5f45c-metrics-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:19.473252 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.473269 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwgwc\" (UniqueName: \"kubernetes.io/projected/b03723b8-b2e9-4f09-9054-c069f1f5f45c-kube-api-access-dwgwc\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:19.599601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.599561 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:19.786975 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.786867 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:19.988788 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.988739 5871 generic.go:356] "Generic (PLEG): container finished" podID="b03723b8-b2e9-4f09-9054-c069f1f5f45c" containerID="3502d501c38037f70e8dfda6944e3aad15054be13451225808c698871a7b146b" exitCode=0 Nov 02 22:33:19.989319 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.988808 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" Nov 02 22:33:19.989319 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.988822 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" event={"ID":"b03723b8-b2e9-4f09-9054-c069f1f5f45c","Type":"ContainerDied","Data":"3502d501c38037f70e8dfda6944e3aad15054be13451225808c698871a7b146b"} Nov 02 22:33:19.989319 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.988944 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7" event={"ID":"b03723b8-b2e9-4f09-9054-c069f1f5f45c","Type":"ContainerDied","Data":"791c11ba52cca852dd74ee11ed9582ac07e48b0bccf2184dac59ec411d412375"} Nov 02 22:33:19.989319 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:19.988970 5871 scope.go:122] "RemoveContainer" containerID="3502d501c38037f70e8dfda6944e3aad15054be13451225808c698871a7b146b" Nov 02 22:33:20.000960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:20.000940 5871 scope.go:122] "RemoveContainer" containerID="3502d501c38037f70e8dfda6944e3aad15054be13451225808c698871a7b146b" Nov 02 22:33:20.001322 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:20.001298 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3502d501c38037f70e8dfda6944e3aad15054be13451225808c698871a7b146b\": container with ID starting with 3502d501c38037f70e8dfda6944e3aad15054be13451225808c698871a7b146b not found: ID does not exist" containerID="3502d501c38037f70e8dfda6944e3aad15054be13451225808c698871a7b146b" Nov 02 22:33:20.001397 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:20.001330 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3502d501c38037f70e8dfda6944e3aad15054be13451225808c698871a7b146b"} err="failed to get container status \"3502d501c38037f70e8dfda6944e3aad15054be13451225808c698871a7b146b\": rpc error: code = NotFound desc = could not find container \"3502d501c38037f70e8dfda6944e3aad15054be13451225808c698871a7b146b\": container with ID starting with 3502d501c38037f70e8dfda6944e3aad15054be13451225808c698871a7b146b not found: ID does not exist" Nov 02 22:33:20.155459 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:20.155423 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7"] Nov 02 22:33:20.164047 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:20.164013 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fe810f-5c9464b7fb-24bz7"] Nov 02 22:33:20.505218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:20.505123 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:20.691032 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:20.690994 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:21.021091 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:21.021051 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:21.215774 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:21.215735 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:21.313974 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:21.313888 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b03723b8-b2e9-4f09-9054-c069f1f5f45c" path="/var/lib/kubelet/pods/b03723b8-b2e9-4f09-9054-c069f1f5f45c/volumes" Nov 02 22:33:21.663054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:21.663016 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:24.971411 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:24.971327 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:25.145119 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:25.145062 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:25.569636 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:25.569594 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:26.206447 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:26.206404 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:26.733719 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:26.733684 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:26.950148 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:26.950115 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:27.040413 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:27.040315 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7"] Nov 02 22:33:27.040594 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:27.040558 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" podUID="01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" containerName="istio-proxy" containerID="cri-o://c3a4067a89c0cdd9b73ffda93e12d0824925e309ac439043c0df2d12cb0522c6" gracePeriod=30 Nov 02 22:33:27.059861 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:27.059825 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5"] Nov 02 22:33:27.060093 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:27.060070 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" podUID="291b0b67-ee70-4d16-ac08-8064e7f0fef1" containerName="istio-proxy" containerID="cri-o://f452c03041a06a856d521de95c28b341058913d28d9663f008cc7f3f5ccb23ca" gracePeriod=30 Nov 02 22:33:27.088503 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:27.088470 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk"] Nov 02 22:33:27.088701 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:27.088679 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" podUID="8c608b09-9551-4d42-91c6-f6cb1a38bab7" containerName="istio-proxy" containerID="cri-o://258d9d53311f4cf9fbd2fd81502d4e949e42a1eed833affbb62ed555fa88b532" gracePeriod=30 Nov 02 22:33:27.132058 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:27.132014 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:27.345930 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:27.345831 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:27.524033 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:27.523991 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:27.698798 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:27.698290 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r"] Nov 02 22:33:27.698798 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:27.698645 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" podUID="dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" containerName="istio-proxy" containerID="cri-o://1c0f9bd93065eca2701608f7d51d24a849b905da08ad4809f623770a0ff73970" gracePeriod=30 Nov 02 22:33:27.703286 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:27.703225 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6"] Nov 02 22:33:27.703451 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:27.703424 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" podUID="667069a7-93cf-42ef-92ec-e825e6f6fabd" containerName="istio-proxy" containerID="cri-o://4f43ede9421c2070255dbf751f1e3b35ef9cdb63ec292b044d570891613817ad" gracePeriod=30 Nov 02 22:33:27.723588 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:27.723556 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:27.910897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:27.910268 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:29.374760 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:29.374724 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:29.558697 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:29.558661 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:29.723514 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:29.723405 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2"] Nov 02 22:33:29.723848 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:29.723737 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" podUID="e364f2d9-c537-4289-b605-547836336c31" containerName="istio-proxy" containerID="cri-o://21f2b004f6461e7a425715bbb1dfadac18c34ebb6183485a9684596f87be956c" gracePeriod=30 Nov 02 22:33:29.726496 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:29.726468 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29"] Nov 02 22:33:29.726713 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:29.726677 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" podUID="2278f047-8cff-4d4f-b799-5b50740987c8" containerName="istio-proxy" containerID="cri-o://7d439cac0be14e875b5917d648619cdb786edc32e7344856911c03c9e28532eb" gracePeriod=30 Nov 02 22:33:29.731887 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:29.731848 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp"] Nov 02 22:33:29.732074 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:29.732054 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" podUID="9384cb9b-0f07-485e-9635-c7dc585a8b76" containerName="istio-proxy" containerID="cri-o://4be053a3f19aa0ebcf22c1068d31aecb03b21ab04ef3181b992b2c0b7156ec5c" gracePeriod=30 Nov 02 22:33:29.738155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:29.738128 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:29.950485 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:29.934003 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:31.457304 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:31.457268 5871 patch_prober.go:29] interesting pod/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:33:31.457692 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:31.457329 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" podUID="667069a7-93cf-42ef-92ec-e825e6f6fabd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:33:31.471578 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:31.471547 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:32.365492 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.365456 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:33:32.395499 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.395460 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:33:32.408242 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.408217 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:33:32.414475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.414445 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-credential-socket\") pod \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " Nov 02 22:33:32.414601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.414497 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-envoy\") pod \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " Nov 02 22:33:32.414601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.414534 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-kube-api-access-rn5bg\" (UniqueName: \"kubernetes.io/projected/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-kube-api-access-rn5bg\") pod \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " Nov 02 22:33:32.414601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.414558 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-workload-socket\") pod \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " Nov 02 22:33:32.414601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.414585 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-envoy\") pod \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " Nov 02 22:33:32.414842 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.414617 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istiod-ca-cert\") pod \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " Nov 02 22:33:32.414842 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.414663 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-workload-certs\") pod \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " Nov 02 22:33:32.414842 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.414706 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-credential-socket" pod "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" (UID: "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:32.414842 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.414720 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/291b0b67-ee70-4d16-ac08-8064e7f0fef1-kube-api-access-cjsnw\" (UniqueName: \"kubernetes.io/projected/291b0b67-ee70-4d16-ac08-8064e7f0fef1-kube-api-access-cjsnw\") pod \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " Nov 02 22:33:32.414842 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.414770 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-token\" (UniqueName: \"kubernetes.io/projected/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-token\") pod \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " Nov 02 22:33:32.414842 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.414830 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-workload-socket\") pod \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " Nov 02 22:33:32.415270 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.414859 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-token\" (UniqueName: \"kubernetes.io/projected/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-token\") pod \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " Nov 02 22:33:32.415270 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.414929 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istiod-ca-cert\") pod \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " Nov 02 22:33:32.415270 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.414941 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-workload-certs" pod "291b0b67-ee70-4d16-ac08-8064e7f0fef1" (UID: "291b0b67-ee70-4d16-ac08-8064e7f0fef1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:32.415270 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.414941 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-workload-socket" pod "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" (UID: "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:32.415270 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.414973 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-data\") pod \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " Nov 02 22:33:32.415270 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.415046 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-podinfo\") pod \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " Nov 02 22:33:32.415270 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.415080 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istiod-ca-cert" pod "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" (UID: "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:32.415270 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.415152 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-credential-socket\") pod \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\" (UID: \"291b0b67-ee70-4d16-ac08-8064e7f0fef1\") " Nov 02 22:33:32.415270 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.415181 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-podinfo\") pod \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " Nov 02 22:33:32.415270 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.415242 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-data\") pod \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " Nov 02 22:33:32.415789 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.415288 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-workload-certs\") pod \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\" (UID: \"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce\") " Nov 02 22:33:32.415789 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.415421 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istiod-ca-cert" pod "291b0b67-ee70-4d16-ac08-8064e7f0fef1" (UID: "291b0b67-ee70-4d16-ac08-8064e7f0fef1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:32.415789 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.415642 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-workload-certs" pod "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" (UID: "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:32.415979 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.415828 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-data" pod "291b0b67-ee70-4d16-ac08-8064e7f0fef1" (UID: "291b0b67-ee70-4d16-ac08-8064e7f0fef1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:32.416944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.416181 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.416944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.416210 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.416944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.416227 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.416944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.416241 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.416944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.416257 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.416944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.416272 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.416944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.416271 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-workload-socket" pod "291b0b67-ee70-4d16-ac08-8064e7f0fef1" (UID: "291b0b67-ee70-4d16-ac08-8064e7f0fef1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:32.416944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.416287 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.417441 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.417241 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-credential-socket" pod "291b0b67-ee70-4d16-ac08-8064e7f0fef1" (UID: "291b0b67-ee70-4d16-ac08-8064e7f0fef1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:32.417441 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.417334 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-data" pod "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" (UID: "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:32.417715 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.417688 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-token" pod "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" (UID: "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:32.417816 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.417778 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-envoy" pod "291b0b67-ee70-4d16-ac08-8064e7f0fef1" (UID: "291b0b67-ee70-4d16-ac08-8064e7f0fef1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:32.417910 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.417842 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-token" pod "291b0b67-ee70-4d16-ac08-8064e7f0fef1" (UID: "291b0b67-ee70-4d16-ac08-8064e7f0fef1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:32.417910 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.417860 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-kube-api-access-rn5bg" pod "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" (UID: "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce"). InnerVolumeSpecName "kube-api-access-rn5bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:32.418064 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.418045 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-envoy" pod "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" (UID: "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:32.418173 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.418153 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-podinfo" pod "291b0b67-ee70-4d16-ac08-8064e7f0fef1" (UID: "291b0b67-ee70-4d16-ac08-8064e7f0fef1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:32.419158 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.419139 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-podinfo" pod "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" (UID: "01dbb32b-088f-4cb3-8c0e-c845fcefc0ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:32.419377 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.419360 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291b0b67-ee70-4d16-ac08-8064e7f0fef1-kube-api-access-cjsnw" pod "291b0b67-ee70-4d16-ac08-8064e7f0fef1" (UID: "291b0b67-ee70-4d16-ac08-8064e7f0fef1"). InnerVolumeSpecName "kube-api-access-cjsnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:32.517591 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.517554 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c608b09-9551-4d42-91c6-f6cb1a38bab7-kube-api-access-jkvg4\" (UniqueName: \"kubernetes.io/projected/8c608b09-9551-4d42-91c6-f6cb1a38bab7-kube-api-access-jkvg4\") pod \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " Nov 02 22:33:32.518161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.517638 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-token\" (UniqueName: \"kubernetes.io/projected/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-token\") pod \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " Nov 02 22:33:32.518161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.517661 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istiod-ca-cert\") pod \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " Nov 02 22:33:32.518161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.517685 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-data\") pod \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " Nov 02 22:33:32.518161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.517712 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-workload-socket\") pod \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " Nov 02 22:33:32.518161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.517751 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-podinfo\") pod \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " Nov 02 22:33:32.518161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.517824 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-workload-certs\") pod \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " Nov 02 22:33:32.518161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.517934 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-envoy\") pod \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " Nov 02 22:33:32.518161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518031 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-credential-socket\") pod \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\" (UID: \"8c608b09-9551-4d42-91c6-f6cb1a38bab7\") " Nov 02 22:33:32.518161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518079 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-data" pod "8c608b09-9551-4d42-91c6-f6cb1a38bab7" (UID: "8c608b09-9551-4d42-91c6-f6cb1a38bab7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:32.518161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518094 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-workload-socket" pod "8c608b09-9551-4d42-91c6-f6cb1a38bab7" (UID: "8c608b09-9551-4d42-91c6-f6cb1a38bab7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:32.518674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518162 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istiod-ca-cert" pod "8c608b09-9551-4d42-91c6-f6cb1a38bab7" (UID: "8c608b09-9551-4d42-91c6-f6cb1a38bab7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:32.518674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518348 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-credential-socket" pod "8c608b09-9551-4d42-91c6-f6cb1a38bab7" (UID: "8c608b09-9551-4d42-91c6-f6cb1a38bab7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:32.518674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518391 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-workload-certs" pod "8c608b09-9551-4d42-91c6-f6cb1a38bab7" (UID: "8c608b09-9551-4d42-91c6-f6cb1a38bab7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:32.518674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518581 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.518674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518602 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjsnw\" (UniqueName: \"kubernetes.io/projected/291b0b67-ee70-4d16-ac08-8064e7f0fef1-kube-api-access-cjsnw\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.518674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518617 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.518674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518631 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.518674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518645 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.518674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518659 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.518674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518673 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.519232 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518688 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.519232 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518702 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.519232 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518720 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.519232 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518736 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.519232 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518750 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.519232 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518765 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.519232 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518780 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.519232 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518795 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn5bg\" (UniqueName: \"kubernetes.io/projected/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce-kube-api-access-rn5bg\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.519232 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.518810 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/291b0b67-ee70-4d16-ac08-8064e7f0fef1-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.520157 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.520090 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-envoy" pod "8c608b09-9551-4d42-91c6-f6cb1a38bab7" (UID: "8c608b09-9551-4d42-91c6-f6cb1a38bab7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:32.520369 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.520327 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-podinfo" pod "8c608b09-9551-4d42-91c6-f6cb1a38bab7" (UID: "8c608b09-9551-4d42-91c6-f6cb1a38bab7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:32.520493 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.520477 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c608b09-9551-4d42-91c6-f6cb1a38bab7-kube-api-access-jkvg4" pod "8c608b09-9551-4d42-91c6-f6cb1a38bab7" (UID: "8c608b09-9551-4d42-91c6-f6cb1a38bab7"). InnerVolumeSpecName "kube-api-access-jkvg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:32.520660 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.520636 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-token" pod "8c608b09-9551-4d42-91c6-f6cb1a38bab7" (UID: "8c608b09-9551-4d42-91c6-f6cb1a38bab7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:32.620199 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.620163 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.620199 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.620196 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.620199 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.620206 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c608b09-9551-4d42-91c6-f6cb1a38bab7-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.620432 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.620215 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkvg4\" (UniqueName: \"kubernetes.io/projected/8c608b09-9551-4d42-91c6-f6cb1a38bab7-kube-api-access-jkvg4\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:32.709547 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.709506 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:33:32.709735 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.709573 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" podUID="9384cb9b-0f07-485e-9635-c7dc585a8b76" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:33:32.952959 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:32.952928 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:33:33.007885 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.007843 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:33:33.026860 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.026821 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-token\" (UniqueName: \"kubernetes.io/projected/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-token\") pod \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " Nov 02 22:33:33.027053 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.026866 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-credential-socket\") pod \"667069a7-93cf-42ef-92ec-e825e6f6fabd\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " Nov 02 22:33:33.027053 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.026965 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-podinfo\") pod \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " Nov 02 22:33:33.027053 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027005 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-workload-certs\") pod \"667069a7-93cf-42ef-92ec-e825e6f6fabd\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " Nov 02 22:33:33.027053 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027043 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667069a7-93cf-42ef-92ec-e825e6f6fabd-kube-api-access-czxhw\" (UniqueName: \"kubernetes.io/projected/667069a7-93cf-42ef-92ec-e825e6f6fabd-kube-api-access-czxhw\") pod \"667069a7-93cf-42ef-92ec-e825e6f6fabd\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " Nov 02 22:33:33.027267 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027168 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-podinfo\") pod \"667069a7-93cf-42ef-92ec-e825e6f6fabd\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " Nov 02 22:33:33.027267 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027224 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-token\" (UniqueName: \"kubernetes.io/projected/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-token\") pod \"667069a7-93cf-42ef-92ec-e825e6f6fabd\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " Nov 02 22:33:33.027267 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027221 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-credential-socket" pod "667069a7-93cf-42ef-92ec-e825e6f6fabd" (UID: "667069a7-93cf-42ef-92ec-e825e6f6fabd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:33.027267 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027259 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-workload-certs" pod "667069a7-93cf-42ef-92ec-e825e6f6fabd" (UID: "667069a7-93cf-42ef-92ec-e825e6f6fabd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:33.027479 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027327 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-workload-socket\") pod \"667069a7-93cf-42ef-92ec-e825e6f6fabd\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " Nov 02 22:33:33.027479 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027386 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-data\") pod \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " Nov 02 22:33:33.027479 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027440 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-envoy\") pod \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " Nov 02 22:33:33.027649 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027486 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-workload-certs\") pod \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " Nov 02 22:33:33.027649 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027559 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/667069a7-93cf-42ef-92ec-e825e6f6fabd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667069a7-93cf-42ef-92ec-e825e6f6fabd-istiod-ca-cert\") pod \"667069a7-93cf-42ef-92ec-e825e6f6fabd\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " Nov 02 22:33:33.027649 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027626 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-data\") pod \"667069a7-93cf-42ef-92ec-e825e6f6fabd\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " Nov 02 22:33:33.027801 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027656 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-envoy\") pod \"667069a7-93cf-42ef-92ec-e825e6f6fabd\" (UID: \"667069a7-93cf-42ef-92ec-e825e6f6fabd\") " Nov 02 22:33:33.027801 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027683 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-data" pod "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" (UID: "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:33.027801 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027699 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-credential-socket\") pod \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " Nov 02 22:33:33.027801 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027686 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-workload-socket" pod "667069a7-93cf-42ef-92ec-e825e6f6fabd" (UID: "667069a7-93cf-42ef-92ec-e825e6f6fabd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:33.027801 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027756 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-kube-api-access-mgbbx\" (UniqueName: \"kubernetes.io/projected/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-kube-api-access-mgbbx\") pod \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " Nov 02 22:33:33.027801 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027789 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-workload-socket\") pod \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " Nov 02 22:33:33.028112 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.027846 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istiod-ca-cert\") pod \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\" (UID: \"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124\") " Nov 02 22:33:33.028112 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.028037 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667069a7-93cf-42ef-92ec-e825e6f6fabd-istiod-ca-cert" pod "667069a7-93cf-42ef-92ec-e825e6f6fabd" (UID: "667069a7-93cf-42ef-92ec-e825e6f6fabd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:33.028372 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.028349 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istiod-ca-cert" pod "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" (UID: "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:33.028566 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.028538 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667069a7-93cf-42ef-92ec-e825e6f6fabd-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.028566 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.028568 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.028758 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.028581 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-data" pod "667069a7-93cf-42ef-92ec-e825e6f6fabd" (UID: "667069a7-93cf-42ef-92ec-e825e6f6fabd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:33.028758 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.028584 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.028758 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.028619 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.028758 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.028636 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.028758 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.028652 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.029824 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.029798 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-token" pod "667069a7-93cf-42ef-92ec-e825e6f6fabd" (UID: "667069a7-93cf-42ef-92ec-e825e6f6fabd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:33.030047 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.030011 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-token" pod "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" (UID: "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:33.030269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.030246 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-workload-certs" pod "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" (UID: "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:33.030373 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.030265 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-podinfo" pod "667069a7-93cf-42ef-92ec-e825e6f6fabd" (UID: "667069a7-93cf-42ef-92ec-e825e6f6fabd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:33.030490 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.030465 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-credential-socket" pod "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" (UID: "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:33.030609 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.030486 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-workload-socket" pod "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" (UID: "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:33.030609 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.030484 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-podinfo" pod "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" (UID: "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:33.030825 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.030803 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-envoy" pod "667069a7-93cf-42ef-92ec-e825e6f6fabd" (UID: "667069a7-93cf-42ef-92ec-e825e6f6fabd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:33.030944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.030920 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667069a7-93cf-42ef-92ec-e825e6f6fabd-kube-api-access-czxhw" pod "667069a7-93cf-42ef-92ec-e825e6f6fabd" (UID: "667069a7-93cf-42ef-92ec-e825e6f6fabd"). InnerVolumeSpecName "kube-api-access-czxhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:33.031381 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.031362 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-envoy" pod "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" (UID: "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:33.031587 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.031570 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-kube-api-access-mgbbx" pod "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" (UID: "dd8e7b9c-7a3a-4c87-b8b9-ab242f263124"). InnerVolumeSpecName "kube-api-access-mgbbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:33.097786 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.097748 5871 generic.go:356] "Generic (PLEG): container finished" podID="8c608b09-9551-4d42-91c6-f6cb1a38bab7" containerID="258d9d53311f4cf9fbd2fd81502d4e949e42a1eed833affbb62ed555fa88b532" exitCode=0 Nov 02 22:33:33.097972 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.097799 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" Nov 02 22:33:33.097972 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.097819 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" event={"ID":"8c608b09-9551-4d42-91c6-f6cb1a38bab7","Type":"ContainerDied","Data":"258d9d53311f4cf9fbd2fd81502d4e949e42a1eed833affbb62ed555fa88b532"} Nov 02 22:33:33.097972 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.097889 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk" event={"ID":"8c608b09-9551-4d42-91c6-f6cb1a38bab7","Type":"ContainerDied","Data":"02bdbb9022c35f62188f6a05839d9b44096b16173e585e177c4bdb113e52fea2"} Nov 02 22:33:33.097972 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.097925 5871 scope.go:122] "RemoveContainer" containerID="258d9d53311f4cf9fbd2fd81502d4e949e42a1eed833affbb62ed555fa88b532" Nov 02 22:33:33.100332 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.100301 5871 generic.go:356] "Generic (PLEG): container finished" podID="291b0b67-ee70-4d16-ac08-8064e7f0fef1" containerID="f452c03041a06a856d521de95c28b341058913d28d9663f008cc7f3f5ccb23ca" exitCode=0 Nov 02 22:33:33.100531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.100393 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" event={"ID":"291b0b67-ee70-4d16-ac08-8064e7f0fef1","Type":"ContainerDied","Data":"f452c03041a06a856d521de95c28b341058913d28d9663f008cc7f3f5ccb23ca"} Nov 02 22:33:33.100531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.100458 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" event={"ID":"291b0b67-ee70-4d16-ac08-8064e7f0fef1","Type":"ContainerDied","Data":"42f78971ec3be67b6feb4f5061b19e3e733c5593dbf6b0d7a2ccc7edf3d03637"} Nov 02 22:33:33.100531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.100468 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5" Nov 02 22:33:33.102321 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.102300 5871 generic.go:356] "Generic (PLEG): container finished" podID="01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" containerID="c3a4067a89c0cdd9b73ffda93e12d0824925e309ac439043c0df2d12cb0522c6" exitCode=0 Nov 02 22:33:33.102527 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.102497 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" event={"ID":"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce","Type":"ContainerDied","Data":"c3a4067a89c0cdd9b73ffda93e12d0824925e309ac439043c0df2d12cb0522c6"} Nov 02 22:33:33.102616 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.102558 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" event={"ID":"01dbb32b-088f-4cb3-8c0e-c845fcefc0ce","Type":"ContainerDied","Data":"c5da3131e5109725cbc713ba65fb414c628df408ea54b51f9fb5953f7531e48d"} Nov 02 22:33:33.102699 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.102643 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7" Nov 02 22:33:33.109994 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.109492 5871 generic.go:356] "Generic (PLEG): container finished" podID="dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" containerID="1c0f9bd93065eca2701608f7d51d24a849b905da08ad4809f623770a0ff73970" exitCode=0 Nov 02 22:33:33.109994 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.109559 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" Nov 02 22:33:33.109994 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.109658 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" event={"ID":"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124","Type":"ContainerDied","Data":"1c0f9bd93065eca2701608f7d51d24a849b905da08ad4809f623770a0ff73970"} Nov 02 22:33:33.109994 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.109727 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r" event={"ID":"dd8e7b9c-7a3a-4c87-b8b9-ab242f263124","Type":"ContainerDied","Data":"eb29c10835c90dca130dcab17c77d957a5528bf8e87e28c75b5b10eaab4790a4"} Nov 02 22:33:33.112139 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.112003 5871 generic.go:356] "Generic (PLEG): container finished" podID="667069a7-93cf-42ef-92ec-e825e6f6fabd" containerID="4f43ede9421c2070255dbf751f1e3b35ef9cdb63ec292b044d570891613817ad" exitCode=0 Nov 02 22:33:33.112139 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.112056 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" event={"ID":"667069a7-93cf-42ef-92ec-e825e6f6fabd","Type":"ContainerDied","Data":"4f43ede9421c2070255dbf751f1e3b35ef9cdb63ec292b044d570891613817ad"} Nov 02 22:33:33.112139 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.112095 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" event={"ID":"667069a7-93cf-42ef-92ec-e825e6f6fabd","Type":"ContainerDied","Data":"437386da1579491052b58192748c4be6d87b8d3544a5514005b2c74782a21234"} Nov 02 22:33:33.112366 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.112169 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6" Nov 02 22:33:33.119605 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.119541 5871 scope.go:122] "RemoveContainer" containerID="258d9d53311f4cf9fbd2fd81502d4e949e42a1eed833affbb62ed555fa88b532" Nov 02 22:33:33.123511 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:33.120199 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258d9d53311f4cf9fbd2fd81502d4e949e42a1eed833affbb62ed555fa88b532\": container with ID starting with 258d9d53311f4cf9fbd2fd81502d4e949e42a1eed833affbb62ed555fa88b532 not found: ID does not exist" containerID="258d9d53311f4cf9fbd2fd81502d4e949e42a1eed833affbb62ed555fa88b532" Nov 02 22:33:33.123511 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.120243 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258d9d53311f4cf9fbd2fd81502d4e949e42a1eed833affbb62ed555fa88b532"} err="failed to get container status \"258d9d53311f4cf9fbd2fd81502d4e949e42a1eed833affbb62ed555fa88b532\": rpc error: code = NotFound desc = could not find container \"258d9d53311f4cf9fbd2fd81502d4e949e42a1eed833affbb62ed555fa88b532\": container with ID starting with 258d9d53311f4cf9fbd2fd81502d4e949e42a1eed833affbb62ed555fa88b532 not found: ID does not exist" Nov 02 22:33:33.123511 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.120269 5871 scope.go:122] "RemoveContainer" containerID="f452c03041a06a856d521de95c28b341058913d28d9663f008cc7f3f5ccb23ca" Nov 02 22:33:33.133642 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.131248 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.133642 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.131288 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.133642 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.131305 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czxhw\" (UniqueName: \"kubernetes.io/projected/667069a7-93cf-42ef-92ec-e825e6f6fabd-kube-api-access-czxhw\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.133642 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.131318 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.133642 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.131332 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.133642 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.131349 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.133642 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.131362 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.133642 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.131375 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.133642 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.131393 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667069a7-93cf-42ef-92ec-e825e6f6fabd-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.133642 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.131406 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.133642 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.131419 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgbbx\" (UniqueName: \"kubernetes.io/projected/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-kube-api-access-mgbbx\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.133642 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.131433 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:33.144929 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.144654 5871 scope.go:122] "RemoveContainer" containerID="6057643f29e05714c7f8cc154f4354fd969ce2d0694bc3aa73440ea0147c3764" Nov 02 22:33:33.160719 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.160691 5871 scope.go:122] "RemoveContainer" containerID="f452c03041a06a856d521de95c28b341058913d28d9663f008cc7f3f5ccb23ca" Nov 02 22:33:33.161105 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:33.161079 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f452c03041a06a856d521de95c28b341058913d28d9663f008cc7f3f5ccb23ca\": container with ID starting with f452c03041a06a856d521de95c28b341058913d28d9663f008cc7f3f5ccb23ca not found: ID does not exist" containerID="f452c03041a06a856d521de95c28b341058913d28d9663f008cc7f3f5ccb23ca" Nov 02 22:33:33.161188 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.161116 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f452c03041a06a856d521de95c28b341058913d28d9663f008cc7f3f5ccb23ca"} err="failed to get container status \"f452c03041a06a856d521de95c28b341058913d28d9663f008cc7f3f5ccb23ca\": rpc error: code = NotFound desc = could not find container \"f452c03041a06a856d521de95c28b341058913d28d9663f008cc7f3f5ccb23ca\": container with ID starting with f452c03041a06a856d521de95c28b341058913d28d9663f008cc7f3f5ccb23ca not found: ID does not exist" Nov 02 22:33:33.161188 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.161135 5871 scope.go:122] "RemoveContainer" containerID="6057643f29e05714c7f8cc154f4354fd969ce2d0694bc3aa73440ea0147c3764" Nov 02 22:33:33.161440 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:33.161409 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6057643f29e05714c7f8cc154f4354fd969ce2d0694bc3aa73440ea0147c3764\": container with ID starting with 6057643f29e05714c7f8cc154f4354fd969ce2d0694bc3aa73440ea0147c3764 not found: ID does not exist" containerID="6057643f29e05714c7f8cc154f4354fd969ce2d0694bc3aa73440ea0147c3764" Nov 02 22:33:33.161544 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.161447 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6057643f29e05714c7f8cc154f4354fd969ce2d0694bc3aa73440ea0147c3764"} err="failed to get container status \"6057643f29e05714c7f8cc154f4354fd969ce2d0694bc3aa73440ea0147c3764\": rpc error: code = NotFound desc = could not find container \"6057643f29e05714c7f8cc154f4354fd969ce2d0694bc3aa73440ea0147c3764\": container with ID starting with 6057643f29e05714c7f8cc154f4354fd969ce2d0694bc3aa73440ea0147c3764 not found: ID does not exist" Nov 02 22:33:33.161544 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.161469 5871 scope.go:122] "RemoveContainer" containerID="c3a4067a89c0cdd9b73ffda93e12d0824925e309ac439043c0df2d12cb0522c6" Nov 02 22:33:33.170781 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.170764 5871 scope.go:122] "RemoveContainer" containerID="c3a4067a89c0cdd9b73ffda93e12d0824925e309ac439043c0df2d12cb0522c6" Nov 02 22:33:33.171127 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:33.171099 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3a4067a89c0cdd9b73ffda93e12d0824925e309ac439043c0df2d12cb0522c6\": container with ID starting with c3a4067a89c0cdd9b73ffda93e12d0824925e309ac439043c0df2d12cb0522c6 not found: ID does not exist" containerID="c3a4067a89c0cdd9b73ffda93e12d0824925e309ac439043c0df2d12cb0522c6" Nov 02 22:33:33.171202 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.171139 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a4067a89c0cdd9b73ffda93e12d0824925e309ac439043c0df2d12cb0522c6"} err="failed to get container status \"c3a4067a89c0cdd9b73ffda93e12d0824925e309ac439043c0df2d12cb0522c6\": rpc error: code = NotFound desc = could not find container \"c3a4067a89c0cdd9b73ffda93e12d0824925e309ac439043c0df2d12cb0522c6\": container with ID starting with c3a4067a89c0cdd9b73ffda93e12d0824925e309ac439043c0df2d12cb0522c6 not found: ID does not exist" Nov 02 22:33:33.171202 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.171159 5871 scope.go:122] "RemoveContainer" containerID="1c0f9bd93065eca2701608f7d51d24a849b905da08ad4809f623770a0ff73970" Nov 02 22:33:33.180559 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.180534 5871 scope.go:122] "RemoveContainer" containerID="1c0f9bd93065eca2701608f7d51d24a849b905da08ad4809f623770a0ff73970" Nov 02 22:33:33.180913 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:33.180885 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0f9bd93065eca2701608f7d51d24a849b905da08ad4809f623770a0ff73970\": container with ID starting with 1c0f9bd93065eca2701608f7d51d24a849b905da08ad4809f623770a0ff73970 not found: ID does not exist" containerID="1c0f9bd93065eca2701608f7d51d24a849b905da08ad4809f623770a0ff73970" Nov 02 22:33:33.180981 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.180924 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0f9bd93065eca2701608f7d51d24a849b905da08ad4809f623770a0ff73970"} err="failed to get container status \"1c0f9bd93065eca2701608f7d51d24a849b905da08ad4809f623770a0ff73970\": rpc error: code = NotFound desc = could not find container \"1c0f9bd93065eca2701608f7d51d24a849b905da08ad4809f623770a0ff73970\": container with ID starting with 1c0f9bd93065eca2701608f7d51d24a849b905da08ad4809f623770a0ff73970 not found: ID does not exist" Nov 02 22:33:33.180981 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.180949 5871 scope.go:122] "RemoveContainer" containerID="4f43ede9421c2070255dbf751f1e3b35ef9cdb63ec292b044d570891613817ad" Nov 02 22:33:33.193150 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.193122 5871 scope.go:122] "RemoveContainer" containerID="4f43ede9421c2070255dbf751f1e3b35ef9cdb63ec292b044d570891613817ad" Nov 02 22:33:33.193570 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:33.193545 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f43ede9421c2070255dbf751f1e3b35ef9cdb63ec292b044d570891613817ad\": container with ID starting with 4f43ede9421c2070255dbf751f1e3b35ef9cdb63ec292b044d570891613817ad not found: ID does not exist" containerID="4f43ede9421c2070255dbf751f1e3b35ef9cdb63ec292b044d570891613817ad" Nov 02 22:33:33.193679 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.193576 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f43ede9421c2070255dbf751f1e3b35ef9cdb63ec292b044d570891613817ad"} err="failed to get container status \"4f43ede9421c2070255dbf751f1e3b35ef9cdb63ec292b044d570891613817ad\": rpc error: code = NotFound desc = could not find container \"4f43ede9421c2070255dbf751f1e3b35ef9cdb63ec292b044d570891613817ad\": container with ID starting with 4f43ede9421c2070255dbf751f1e3b35ef9cdb63ec292b044d570891613817ad not found: ID does not exist" Nov 02 22:33:33.313597 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.313564 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5"] Nov 02 22:33:33.317485 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.317456 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-l5xd5"] Nov 02 22:33:33.434111 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.434029 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7"] Nov 02 22:33:33.437482 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.437453 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-h5tg7"] Nov 02 22:33:33.547931 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.547865 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r"] Nov 02 22:33:33.553070 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.553040 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-m5b8r"] Nov 02 22:33:33.663412 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.663381 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk"] Nov 02 22:33:33.671130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.671096 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-49a557-openshift-default-5796bb9575-65ngk"] Nov 02 22:33:33.783337 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.783241 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6"] Nov 02 22:33:33.786556 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:33.786526 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-98a222-openshift-default-587d758db7-v9st6"] Nov 02 22:33:34.172418 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:34.172378 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:33:34.172592 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:34.172445 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" podUID="e364f2d9-c537-4289-b605-547836336c31" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:33:35.132907 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.132848 5871 patch_prober.go:29] interesting pod/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:33:35.133277 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.132929 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" podUID="2278f047-8cff-4d4f-b799-5b50740987c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:33:35.313238 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.313193 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" path="/var/lib/kubelet/pods/01dbb32b-088f-4cb3-8c0e-c845fcefc0ce/volumes" Nov 02 22:33:35.313706 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.313692 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="291b0b67-ee70-4d16-ac08-8064e7f0fef1" path="/var/lib/kubelet/pods/291b0b67-ee70-4d16-ac08-8064e7f0fef1/volumes" Nov 02 22:33:35.314148 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.314119 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="667069a7-93cf-42ef-92ec-e825e6f6fabd" path="/var/lib/kubelet/pods/667069a7-93cf-42ef-92ec-e825e6f6fabd/volumes" Nov 02 22:33:35.315250 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.315225 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c608b09-9551-4d42-91c6-f6cb1a38bab7" path="/var/lib/kubelet/pods/8c608b09-9551-4d42-91c6-f6cb1a38bab7/volumes" Nov 02 22:33:35.315776 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.315756 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" path="/var/lib/kubelet/pods/dd8e7b9c-7a3a-4c87-b8b9-ab242f263124/volumes" Nov 02 22:33:35.529961 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.529927 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:35.715506 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.715482 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:33:35.718897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.718855 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:33:35.761469 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.761431 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-istio-envoy\") pod \"e364f2d9-c537-4289-b605-547836336c31\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " Nov 02 22:33:35.761669 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.761487 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-workload-certs\") pod \"9384cb9b-0f07-485e-9635-c7dc585a8b76\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " Nov 02 22:33:35.761669 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.761507 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-envoy\") pod \"9384cb9b-0f07-485e-9635-c7dc585a8b76\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " Nov 02 22:33:35.761669 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.761525 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-istio-data\") pod \"e364f2d9-c537-4289-b605-547836336c31\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " Nov 02 22:33:35.761669 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.761547 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-workload-socket\") pod \"9384cb9b-0f07-485e-9635-c7dc585a8b76\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " Nov 02 22:33:35.761669 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.761585 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-podinfo\") pod \"9384cb9b-0f07-485e-9635-c7dc585a8b76\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " Nov 02 22:33:35.761669 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.761615 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-data\") pod \"9384cb9b-0f07-485e-9635-c7dc585a8b76\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " Nov 02 22:33:35.762042 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.761684 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e364f2d9-c537-4289-b605-547836336c31-kube-api-access-h5tdf\" (UniqueName: \"kubernetes.io/projected/e364f2d9-c537-4289-b605-547836336c31-kube-api-access-h5tdf\") pod \"e364f2d9-c537-4289-b605-547836336c31\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " Nov 02 22:33:35.762042 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.761714 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e364f2d9-c537-4289-b605-547836336c31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e364f2d9-c537-4289-b605-547836336c31-istiod-ca-cert\") pod \"e364f2d9-c537-4289-b605-547836336c31\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " Nov 02 22:33:35.762042 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.761804 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-workload-certs\") pod \"e364f2d9-c537-4289-b605-547836336c31\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " Nov 02 22:33:35.762042 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.761842 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e364f2d9-c537-4289-b605-547836336c31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e364f2d9-c537-4289-b605-547836336c31-istio-podinfo\") pod \"e364f2d9-c537-4289-b605-547836336c31\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " Nov 02 22:33:35.762042 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.761868 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9384cb9b-0f07-485e-9635-c7dc585a8b76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9384cb9b-0f07-485e-9635-c7dc585a8b76-istiod-ca-cert\") pod \"9384cb9b-0f07-485e-9635-c7dc585a8b76\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " Nov 02 22:33:35.762042 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.761911 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-credential-socket\") pod \"e364f2d9-c537-4289-b605-547836336c31\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " Nov 02 22:33:35.762042 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.761947 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e364f2d9-c537-4289-b605-547836336c31-istio-token\" (UniqueName: \"kubernetes.io/projected/e364f2d9-c537-4289-b605-547836336c31-istio-token\") pod \"e364f2d9-c537-4289-b605-547836336c31\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " Nov 02 22:33:35.762042 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.761986 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9384cb9b-0f07-485e-9635-c7dc585a8b76-kube-api-access-p8628\" (UniqueName: \"kubernetes.io/projected/9384cb9b-0f07-485e-9635-c7dc585a8b76-kube-api-access-p8628\") pod \"9384cb9b-0f07-485e-9635-c7dc585a8b76\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " Nov 02 22:33:35.762042 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.761995 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-data" pod "9384cb9b-0f07-485e-9635-c7dc585a8b76" (UID: "9384cb9b-0f07-485e-9635-c7dc585a8b76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:35.762042 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.762005 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-workload-socket" pod "9384cb9b-0f07-485e-9635-c7dc585a8b76" (UID: "9384cb9b-0f07-485e-9635-c7dc585a8b76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:35.762531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.762106 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-credential-socket\") pod \"9384cb9b-0f07-485e-9635-c7dc585a8b76\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " Nov 02 22:33:35.762531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.762158 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-token\" (UniqueName: \"kubernetes.io/projected/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-token\") pod \"9384cb9b-0f07-485e-9635-c7dc585a8b76\" (UID: \"9384cb9b-0f07-485e-9635-c7dc585a8b76\") " Nov 02 22:33:35.762531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.762206 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-workload-socket\") pod \"e364f2d9-c537-4289-b605-547836336c31\" (UID: \"e364f2d9-c537-4289-b605-547836336c31\") " Nov 02 22:33:35.762531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.762211 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-istio-data" pod "e364f2d9-c537-4289-b605-547836336c31" (UID: "e364f2d9-c537-4289-b605-547836336c31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:35.762531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.762265 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-credential-socket" pod "e364f2d9-c537-4289-b605-547836336c31" (UID: "e364f2d9-c537-4289-b605-547836336c31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:35.762531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.762484 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-workload-certs" pod "9384cb9b-0f07-485e-9635-c7dc585a8b76" (UID: "9384cb9b-0f07-485e-9635-c7dc585a8b76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:35.762531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.762502 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9384cb9b-0f07-485e-9635-c7dc585a8b76-istiod-ca-cert" pod "9384cb9b-0f07-485e-9635-c7dc585a8b76" (UID: "9384cb9b-0f07-485e-9635-c7dc585a8b76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:35.762918 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.762713 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.762918 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.762735 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.762918 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.762750 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.762918 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.762763 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.762918 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.762776 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9384cb9b-0f07-485e-9635-c7dc585a8b76-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.762918 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.762789 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.762918 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.762803 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e364f2d9-c537-4289-b605-547836336c31-istiod-ca-cert" pod "e364f2d9-c537-4289-b605-547836336c31" (UID: "e364f2d9-c537-4289-b605-547836336c31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:35.763276 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.763029 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-workload-socket" pod "e364f2d9-c537-4289-b605-547836336c31" (UID: "e364f2d9-c537-4289-b605-547836336c31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:35.763276 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.763214 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-credential-socket" pod "9384cb9b-0f07-485e-9635-c7dc585a8b76" (UID: "9384cb9b-0f07-485e-9635-c7dc585a8b76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:35.764104 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.764077 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-envoy" pod "9384cb9b-0f07-485e-9635-c7dc585a8b76" (UID: "9384cb9b-0f07-485e-9635-c7dc585a8b76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:35.764275 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.764251 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-istio-envoy" pod "e364f2d9-c537-4289-b605-547836336c31" (UID: "e364f2d9-c537-4289-b605-547836336c31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:35.764407 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.764378 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-workload-certs" pod "e364f2d9-c537-4289-b605-547836336c31" (UID: "e364f2d9-c537-4289-b605-547836336c31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:35.764714 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.764696 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e364f2d9-c537-4289-b605-547836336c31-istio-podinfo" pod "e364f2d9-c537-4289-b605-547836336c31" (UID: "e364f2d9-c537-4289-b605-547836336c31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:35.764810 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.764775 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9384cb9b-0f07-485e-9635-c7dc585a8b76-kube-api-access-p8628" pod "9384cb9b-0f07-485e-9635-c7dc585a8b76" (UID: "9384cb9b-0f07-485e-9635-c7dc585a8b76"). InnerVolumeSpecName "kube-api-access-p8628". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:35.764810 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.764799 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-token" pod "9384cb9b-0f07-485e-9635-c7dc585a8b76" (UID: "9384cb9b-0f07-485e-9635-c7dc585a8b76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:35.765151 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.765121 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-podinfo" pod "9384cb9b-0f07-485e-9635-c7dc585a8b76" (UID: "9384cb9b-0f07-485e-9635-c7dc585a8b76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:35.765264 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.765248 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e364f2d9-c537-4289-b605-547836336c31-istio-token" pod "e364f2d9-c537-4289-b605-547836336c31" (UID: "e364f2d9-c537-4289-b605-547836336c31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:35.765883 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.765859 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e364f2d9-c537-4289-b605-547836336c31-kube-api-access-h5tdf" pod "e364f2d9-c537-4289-b605-547836336c31" (UID: "e364f2d9-c537-4289-b605-547836336c31"). InnerVolumeSpecName "kube-api-access-h5tdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:35.864353 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.864308 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8628\" (UniqueName: \"kubernetes.io/projected/9384cb9b-0f07-485e-9635-c7dc585a8b76-kube-api-access-p8628\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.864353 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.864346 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.864353 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.864356 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.864606 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.864365 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.864606 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.864375 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.864606 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.864386 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.864606 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.864395 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9384cb9b-0f07-485e-9635-c7dc585a8b76-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.864606 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.864404 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5tdf\" (UniqueName: \"kubernetes.io/projected/e364f2d9-c537-4289-b605-547836336c31-kube-api-access-h5tdf\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.864606 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.864414 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e364f2d9-c537-4289-b605-547836336c31-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.864606 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.864424 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e364f2d9-c537-4289-b605-547836336c31-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.864606 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.864433 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e364f2d9-c537-4289-b605-547836336c31-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.864606 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:35.864442 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e364f2d9-c537-4289-b605-547836336c31-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.083531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.083503 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:33:36.138385 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.138347 5871 generic.go:356] "Generic (PLEG): container finished" podID="9384cb9b-0f07-485e-9635-c7dc585a8b76" containerID="4be053a3f19aa0ebcf22c1068d31aecb03b21ab04ef3181b992b2c0b7156ec5c" exitCode=0 Nov 02 22:33:36.138385 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.138389 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" Nov 02 22:33:36.138889 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.138436 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" event={"ID":"9384cb9b-0f07-485e-9635-c7dc585a8b76","Type":"ContainerDied","Data":"4be053a3f19aa0ebcf22c1068d31aecb03b21ab04ef3181b992b2c0b7156ec5c"} Nov 02 22:33:36.138889 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.138488 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp" event={"ID":"9384cb9b-0f07-485e-9635-c7dc585a8b76","Type":"ContainerDied","Data":"164e8159b18d8ed2789ca3ae3bd3580e94ed010434a2d2095301ee497503b554"} Nov 02 22:33:36.138889 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.138513 5871 scope.go:122] "RemoveContainer" containerID="4be053a3f19aa0ebcf22c1068d31aecb03b21ab04ef3181b992b2c0b7156ec5c" Nov 02 22:33:36.140005 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.139977 5871 generic.go:356] "Generic (PLEG): container finished" podID="e364f2d9-c537-4289-b605-547836336c31" containerID="21f2b004f6461e7a425715bbb1dfadac18c34ebb6183485a9684596f87be956c" exitCode=0 Nov 02 22:33:36.140126 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.140023 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" Nov 02 22:33:36.140126 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.140058 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" event={"ID":"e364f2d9-c537-4289-b605-547836336c31","Type":"ContainerDied","Data":"21f2b004f6461e7a425715bbb1dfadac18c34ebb6183485a9684596f87be956c"} Nov 02 22:33:36.140126 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.140110 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2" event={"ID":"e364f2d9-c537-4289-b605-547836336c31","Type":"ContainerDied","Data":"c851d59e12b590aa24cc2d84a93d5cf0a4bccbc1e47629c518154b476d685811"} Nov 02 22:33:36.141597 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.141572 5871 generic.go:356] "Generic (PLEG): container finished" podID="2278f047-8cff-4d4f-b799-5b50740987c8" containerID="7d439cac0be14e875b5917d648619cdb786edc32e7344856911c03c9e28532eb" exitCode=0 Nov 02 22:33:36.141788 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.141766 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" Nov 02 22:33:36.141868 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.141614 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" event={"ID":"2278f047-8cff-4d4f-b799-5b50740987c8","Type":"ContainerDied","Data":"7d439cac0be14e875b5917d648619cdb786edc32e7344856911c03c9e28532eb"} Nov 02 22:33:36.141946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.141887 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29" event={"ID":"2278f047-8cff-4d4f-b799-5b50740987c8","Type":"ContainerDied","Data":"97f56dc38a202ccec3d48c5c6c27a91ac22d36b8c7527d563d42281b1514621a"} Nov 02 22:33:36.149513 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.149489 5871 scope.go:122] "RemoveContainer" containerID="4be053a3f19aa0ebcf22c1068d31aecb03b21ab04ef3181b992b2c0b7156ec5c" Nov 02 22:33:36.149983 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:36.149956 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be053a3f19aa0ebcf22c1068d31aecb03b21ab04ef3181b992b2c0b7156ec5c\": container with ID starting with 4be053a3f19aa0ebcf22c1068d31aecb03b21ab04ef3181b992b2c0b7156ec5c not found: ID does not exist" containerID="4be053a3f19aa0ebcf22c1068d31aecb03b21ab04ef3181b992b2c0b7156ec5c" Nov 02 22:33:36.150071 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.149998 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be053a3f19aa0ebcf22c1068d31aecb03b21ab04ef3181b992b2c0b7156ec5c"} err="failed to get container status \"4be053a3f19aa0ebcf22c1068d31aecb03b21ab04ef3181b992b2c0b7156ec5c\": rpc error: code = NotFound desc = could not find container \"4be053a3f19aa0ebcf22c1068d31aecb03b21ab04ef3181b992b2c0b7156ec5c\": container with ID starting with 4be053a3f19aa0ebcf22c1068d31aecb03b21ab04ef3181b992b2c0b7156ec5c not found: ID does not exist" Nov 02 22:33:36.150071 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.150034 5871 scope.go:122] "RemoveContainer" containerID="21f2b004f6461e7a425715bbb1dfadac18c34ebb6183485a9684596f87be956c" Nov 02 22:33:36.160026 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.160001 5871 scope.go:122] "RemoveContainer" containerID="21f2b004f6461e7a425715bbb1dfadac18c34ebb6183485a9684596f87be956c" Nov 02 22:33:36.160365 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:36.160344 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f2b004f6461e7a425715bbb1dfadac18c34ebb6183485a9684596f87be956c\": container with ID starting with 21f2b004f6461e7a425715bbb1dfadac18c34ebb6183485a9684596f87be956c not found: ID does not exist" containerID="21f2b004f6461e7a425715bbb1dfadac18c34ebb6183485a9684596f87be956c" Nov 02 22:33:36.160413 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.160375 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f2b004f6461e7a425715bbb1dfadac18c34ebb6183485a9684596f87be956c"} err="failed to get container status \"21f2b004f6461e7a425715bbb1dfadac18c34ebb6183485a9684596f87be956c\": rpc error: code = NotFound desc = could not find container \"21f2b004f6461e7a425715bbb1dfadac18c34ebb6183485a9684596f87be956c\": container with ID starting with 21f2b004f6461e7a425715bbb1dfadac18c34ebb6183485a9684596f87be956c not found: ID does not exist" Nov 02 22:33:36.160413 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.160396 5871 scope.go:122] "RemoveContainer" containerID="7d439cac0be14e875b5917d648619cdb786edc32e7344856911c03c9e28532eb" Nov 02 22:33:36.167703 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.167676 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-credential-socket\") pod \"2278f047-8cff-4d4f-b799-5b50740987c8\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " Nov 02 22:33:36.167798 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.167731 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2278f047-8cff-4d4f-b799-5b50740987c8-kube-api-access-5rrqs\" (UniqueName: \"kubernetes.io/projected/2278f047-8cff-4d4f-b799-5b50740987c8-kube-api-access-5rrqs\") pod \"2278f047-8cff-4d4f-b799-5b50740987c8\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " Nov 02 22:33:36.167798 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.167760 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2278f047-8cff-4d4f-b799-5b50740987c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2278f047-8cff-4d4f-b799-5b50740987c8-istio-podinfo\") pod \"2278f047-8cff-4d4f-b799-5b50740987c8\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " Nov 02 22:33:36.167913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.167851 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2278f047-8cff-4d4f-b799-5b50740987c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2278f047-8cff-4d4f-b799-5b50740987c8-istiod-ca-cert\") pod \"2278f047-8cff-4d4f-b799-5b50740987c8\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " Nov 02 22:33:36.167964 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.167911 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-istio-data\") pod \"2278f047-8cff-4d4f-b799-5b50740987c8\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " Nov 02 22:33:36.168029 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.167969 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2278f047-8cff-4d4f-b799-5b50740987c8-istio-token\" (UniqueName: \"kubernetes.io/projected/2278f047-8cff-4d4f-b799-5b50740987c8-istio-token\") pod \"2278f047-8cff-4d4f-b799-5b50740987c8\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " Nov 02 22:33:36.168029 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.167982 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-credential-socket" pod "2278f047-8cff-4d4f-b799-5b50740987c8" (UID: "2278f047-8cff-4d4f-b799-5b50740987c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:36.168029 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.168004 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-workload-certs\") pod \"2278f047-8cff-4d4f-b799-5b50740987c8\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " Nov 02 22:33:36.168180 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.168040 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-istio-envoy\") pod \"2278f047-8cff-4d4f-b799-5b50740987c8\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " Nov 02 22:33:36.168180 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.168093 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-workload-socket\") pod \"2278f047-8cff-4d4f-b799-5b50740987c8\" (UID: \"2278f047-8cff-4d4f-b799-5b50740987c8\") " Nov 02 22:33:36.168376 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.168264 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2278f047-8cff-4d4f-b799-5b50740987c8-istiod-ca-cert" pod "2278f047-8cff-4d4f-b799-5b50740987c8" (UID: "2278f047-8cff-4d4f-b799-5b50740987c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:36.168440 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.168407 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-workload-certs" pod "2278f047-8cff-4d4f-b799-5b50740987c8" (UID: "2278f047-8cff-4d4f-b799-5b50740987c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:36.168496 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.168457 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-istio-data" pod "2278f047-8cff-4d4f-b799-5b50740987c8" (UID: "2278f047-8cff-4d4f-b799-5b50740987c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:36.168586 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.168560 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-workload-socket" pod "2278f047-8cff-4d4f-b799-5b50740987c8" (UID: "2278f047-8cff-4d4f-b799-5b50740987c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:36.168753 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.168733 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.168812 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.168759 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.168812 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.168776 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.168812 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.168791 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2278f047-8cff-4d4f-b799-5b50740987c8-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.168812 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.168804 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.169944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.169924 5871 scope.go:122] "RemoveContainer" containerID="7d439cac0be14e875b5917d648619cdb786edc32e7344856911c03c9e28532eb" Nov 02 22:33:36.170248 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.170215 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2278f047-8cff-4d4f-b799-5b50740987c8-istio-podinfo" pod "2278f047-8cff-4d4f-b799-5b50740987c8" (UID: "2278f047-8cff-4d4f-b799-5b50740987c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:33:36.170409 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:36.170382 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d439cac0be14e875b5917d648619cdb786edc32e7344856911c03c9e28532eb\": container with ID starting with 7d439cac0be14e875b5917d648619cdb786edc32e7344856911c03c9e28532eb not found: ID does not exist" containerID="7d439cac0be14e875b5917d648619cdb786edc32e7344856911c03c9e28532eb" Nov 02 22:33:36.170505 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.170415 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d439cac0be14e875b5917d648619cdb786edc32e7344856911c03c9e28532eb"} err="failed to get container status \"7d439cac0be14e875b5917d648619cdb786edc32e7344856911c03c9e28532eb\": rpc error: code = NotFound desc = could not find container \"7d439cac0be14e875b5917d648619cdb786edc32e7344856911c03c9e28532eb\": container with ID starting with 7d439cac0be14e875b5917d648619cdb786edc32e7344856911c03c9e28532eb not found: ID does not exist" Nov 02 22:33:36.170575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.170552 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-istio-envoy" pod "2278f047-8cff-4d4f-b799-5b50740987c8" (UID: "2278f047-8cff-4d4f-b799-5b50740987c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:33:36.170645 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.170630 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2278f047-8cff-4d4f-b799-5b50740987c8-istio-token" pod "2278f047-8cff-4d4f-b799-5b50740987c8" (UID: "2278f047-8cff-4d4f-b799-5b50740987c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:36.170686 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.170669 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2278f047-8cff-4d4f-b799-5b50740987c8-kube-api-access-5rrqs" pod "2278f047-8cff-4d4f-b799-5b50740987c8" (UID: "2278f047-8cff-4d4f-b799-5b50740987c8"). InnerVolumeSpecName "kube-api-access-5rrqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:36.270431 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.270341 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2278f047-8cff-4d4f-b799-5b50740987c8-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.270431 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.270376 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2278f047-8cff-4d4f-b799-5b50740987c8-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.270431 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.270413 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rrqs\" (UniqueName: \"kubernetes.io/projected/2278f047-8cff-4d4f-b799-5b50740987c8-kube-api-access-5rrqs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.270431 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.270425 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2278f047-8cff-4d4f-b799-5b50740987c8-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:36.296542 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.296487 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:36.323940 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.323901 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp"] Nov 02 22:33:36.327598 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.327553 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-zb8tp"] Nov 02 22:33:36.436813 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.436778 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2"] Nov 02 22:33:36.440207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.440182 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-6lvf2"] Nov 02 22:33:36.604407 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.604322 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29"] Nov 02 22:33:36.608305 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:36.608282 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe810f-openshift-default-86c89d8ff9-tdn29"] Nov 02 22:33:37.313404 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:37.313367 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2278f047-8cff-4d4f-b799-5b50740987c8" path="/var/lib/kubelet/pods/2278f047-8cff-4d4f-b799-5b50740987c8/volumes" Nov 02 22:33:37.313854 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:37.313840 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9384cb9b-0f07-485e-9635-c7dc585a8b76" path="/var/lib/kubelet/pods/9384cb9b-0f07-485e-9635-c7dc585a8b76/volumes" Nov 02 22:33:37.314274 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:37.314261 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e364f2d9-c537-4289-b605-547836336c31" path="/var/lib/kubelet/pods/e364f2d9-c537-4289-b605-547836336c31/volumes" Nov 02 22:33:38.191424 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:38.191389 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:41.063555 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:41.063521 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:42.334792 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:42.334755 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:43.771082 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:33:43.771039 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:33:45.879687 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:45.879653 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:46.140155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:46.140076 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:47.258802 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:47.258765 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:33:50.090744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:50.090706 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:50.692207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:50.692169 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:54.207409 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:54.207370 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:55.866301 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:55.866259 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:33:56.583862 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:33:56.583828 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:00.778745 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:00.778708 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:03.714374 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:03.714335 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:03.901584 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:03.901547 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:04.832631 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:04.832594 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8"] Nov 02 22:34:04.833064 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:04.832823 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" podUID="7c7a6cfc-826c-4c21-8d77-35272c34d117" containerName="maas-api" containerID="cri-o://67d4f5649c2aa65fd9fc77e0508b41472f00baf1eaad337469414e6640a78683" gracePeriod=30 Nov 02 22:34:04.959220 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:04.959180 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:05.121052 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.121022 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:34:05.194566 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.194521 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c7a6cfc-826c-4c21-8d77-35272c34d117-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7a6cfc-826c-4c21-8d77-35272c34d117-maas-api-tls\") pod \"7c7a6cfc-826c-4c21-8d77-35272c34d117\" (UID: \"7c7a6cfc-826c-4c21-8d77-35272c34d117\") " Nov 02 22:34:05.194762 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.194623 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7a6cfc-826c-4c21-8d77-35272c34d117-kube-api-access-7h5z2\" (UniqueName: \"kubernetes.io/projected/7c7a6cfc-826c-4c21-8d77-35272c34d117-kube-api-access-7h5z2\") pod \"7c7a6cfc-826c-4c21-8d77-35272c34d117\" (UID: \"7c7a6cfc-826c-4c21-8d77-35272c34d117\") " Nov 02 22:34:05.194815 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.194785 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7a6cfc-826c-4c21-8d77-35272c34d117-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7a6cfc-826c-4c21-8d77-35272c34d117-openshift-service-ca\") pod \"7c7a6cfc-826c-4c21-8d77-35272c34d117\" (UID: \"7c7a6cfc-826c-4c21-8d77-35272c34d117\") " Nov 02 22:34:05.194971 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.194951 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c7a6cfc-826c-4c21-8d77-35272c34d117-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7a6cfc-826c-4c21-8d77-35272c34d117-metrics-certs\") pod \"7c7a6cfc-826c-4c21-8d77-35272c34d117\" (UID: \"7c7a6cfc-826c-4c21-8d77-35272c34d117\") " Nov 02 22:34:05.195199 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.195172 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7a6cfc-826c-4c21-8d77-35272c34d117-openshift-service-ca" pod "7c7a6cfc-826c-4c21-8d77-35272c34d117" (UID: "7c7a6cfc-826c-4c21-8d77-35272c34d117"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:34:05.195458 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.195437 5871 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c7a6cfc-826c-4c21-8d77-35272c34d117-openshift-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:05.196866 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.196837 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7a6cfc-826c-4c21-8d77-35272c34d117-kube-api-access-7h5z2" pod "7c7a6cfc-826c-4c21-8d77-35272c34d117" (UID: "7c7a6cfc-826c-4c21-8d77-35272c34d117"). InnerVolumeSpecName "kube-api-access-7h5z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:05.196972 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.196843 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7a6cfc-826c-4c21-8d77-35272c34d117-maas-api-tls" pod "7c7a6cfc-826c-4c21-8d77-35272c34d117" (UID: "7c7a6cfc-826c-4c21-8d77-35272c34d117"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:05.196972 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.196901 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7a6cfc-826c-4c21-8d77-35272c34d117-metrics-certs" pod "7c7a6cfc-826c-4c21-8d77-35272c34d117" (UID: "7c7a6cfc-826c-4c21-8d77-35272c34d117"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:05.296851 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.296808 5871 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c7a6cfc-826c-4c21-8d77-35272c34d117-maas-api-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:05.296851 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.296846 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h5z2\" (UniqueName: \"kubernetes.io/projected/7c7a6cfc-826c-4c21-8d77-35272c34d117-kube-api-access-7h5z2\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:05.296851 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.296856 5871 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c7a6cfc-826c-4c21-8d77-35272c34d117-metrics-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:05.323756 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.323727 5871 generic.go:356] "Generic (PLEG): container finished" podID="7c7a6cfc-826c-4c21-8d77-35272c34d117" containerID="67d4f5649c2aa65fd9fc77e0508b41472f00baf1eaad337469414e6640a78683" exitCode=0 Nov 02 22:34:05.323928 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.323770 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" Nov 02 22:34:05.323928 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.323808 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" event={"ID":"7c7a6cfc-826c-4c21-8d77-35272c34d117","Type":"ContainerDied","Data":"67d4f5649c2aa65fd9fc77e0508b41472f00baf1eaad337469414e6640a78683"} Nov 02 22:34:05.323928 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.323858 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8" event={"ID":"7c7a6cfc-826c-4c21-8d77-35272c34d117","Type":"ContainerDied","Data":"3c4aa5d42797657e1edf78eb3db14aa8035f660cd0ca9934b763717793f145de"} Nov 02 22:34:05.323928 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.323895 5871 scope.go:122] "RemoveContainer" containerID="67d4f5649c2aa65fd9fc77e0508b41472f00baf1eaad337469414e6640a78683" Nov 02 22:34:05.333861 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.333840 5871 scope.go:122] "RemoveContainer" containerID="67d4f5649c2aa65fd9fc77e0508b41472f00baf1eaad337469414e6640a78683" Nov 02 22:34:05.334208 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:34:05.334190 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d4f5649c2aa65fd9fc77e0508b41472f00baf1eaad337469414e6640a78683\": container with ID starting with 67d4f5649c2aa65fd9fc77e0508b41472f00baf1eaad337469414e6640a78683 not found: ID does not exist" containerID="67d4f5649c2aa65fd9fc77e0508b41472f00baf1eaad337469414e6640a78683" Nov 02 22:34:05.334256 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.334219 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d4f5649c2aa65fd9fc77e0508b41472f00baf1eaad337469414e6640a78683"} err="failed to get container status \"67d4f5649c2aa65fd9fc77e0508b41472f00baf1eaad337469414e6640a78683\": rpc error: code = NotFound desc = could not find container \"67d4f5649c2aa65fd9fc77e0508b41472f00baf1eaad337469414e6640a78683\": container with ID starting with 67d4f5649c2aa65fd9fc77e0508b41472f00baf1eaad337469414e6640a78683 not found: ID does not exist" Nov 02 22:34:05.490126 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.490084 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8"] Nov 02 22:34:05.496001 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:05.495964 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb1a7ce7-f4bc695f4-bfxn8"] Nov 02 22:34:06.769058 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:06.769014 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:07.200338 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:07.200303 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:07.313577 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:07.313542 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7a6cfc-826c-4c21-8d77-35272c34d117" path="/var/lib/kubelet/pods/7c7a6cfc-826c-4c21-8d77-35272c34d117/volumes" Nov 02 22:34:11.467560 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:11.467477 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:34:11.673048 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:11.673017 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:13.771202 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:34:13.771166 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:34:17.354479 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:17.354446 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:17.579035 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:17.578991 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:17.902533 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:17.902499 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:18.320422 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:18.320302 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:22.697822 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:22.697786 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:24.158090 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:24.158055 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:24.336073 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:24.335985 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:24.534502 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:24.534390 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:24.981181 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:24.981132 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:27.785360 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:27.785324 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:29.910131 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:29.910094 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz"] Nov 02 22:34:29.960071 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:29.960027 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p"] Nov 02 22:34:30.107635 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:30.107597 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:32.973750 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:32.973712 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:37.009866 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:37.009825 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47"] Nov 02 22:34:37.296835 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:37.296754 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:37.471328 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:37.471292 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:37.843974 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:37.843910 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:42.010476 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:42.010393 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" podUID="e449a219-b519-48b7-a4b0-b9dc589cdeb3" containerName="payload-pre-processing" containerID="cri-o://856366419bb8331212bf34e7fd673fb0f2108883d28eda984a35687f02bdcf72" gracePeriod=25 Nov 02 22:34:42.654394 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:42.654357 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:43.771889 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:34:43.771835 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:34:44.298577 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:44.298531 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" podUID="e449a219-b519-48b7-a4b0-b9dc589cdeb3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:34:44.931839 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:44.931796 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" containerID="cri-o://c5859f1b20c931e286714f4815b82f7b86b1665c5cbd617df273cb8b5264c760" gracePeriod=45 Nov 02 22:34:44.980242 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:44.980204 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" containerID="cri-o://7a9de685be59f2fbcedf0a63f9228f69e803cafea059231566802828ebe114d6" gracePeriod=45 Nov 02 22:34:45.017103 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.017068 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": dial tcp 10.134.0.72:8000: connect: connection refused" Nov 02 22:34:45.068694 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.068656 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused" Nov 02 22:34:45.253969 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.253939 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:34:45.277967 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.277940 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:34:45.341423 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.341381 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-dshm\") pod \"5d76402c-4d39-4c03-8018-16f2b97b5fab\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " Nov 02 22:34:45.341602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.341437 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-model-cache\") pod \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " Nov 02 22:34:45.341602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.341481 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-model-cache\") pod \"5d76402c-4d39-4c03-8018-16f2b97b5fab\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " Nov 02 22:34:45.341602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.341504 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-tmp-dir\") pod \"5d76402c-4d39-4c03-8018-16f2b97b5fab\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " Nov 02 22:34:45.341602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.341550 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-tmp-dir\") pod \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " Nov 02 22:34:45.341602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.341581 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d76402c-4d39-4c03-8018-16f2b97b5fab-kube-api-access-pmk8w\" (UniqueName: \"kubernetes.io/projected/5d76402c-4d39-4c03-8018-16f2b97b5fab-kube-api-access-pmk8w\") pod \"5d76402c-4d39-4c03-8018-16f2b97b5fab\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " Nov 02 22:34:45.341893 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.341614 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-home\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-home\") pod \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " Nov 02 22:34:45.341893 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.341809 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-model-cache" pod "5d76402c-4d39-4c03-8018-16f2b97b5fab" (UID: "5d76402c-4d39-4c03-8018-16f2b97b5fab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:45.341893 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.341815 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-model-cache" pod "a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" (UID: "a1d285f6-5118-41bc-aa3a-0c3348c8d1b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:45.341893 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.341795 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-tmp-dir" pod "5d76402c-4d39-4c03-8018-16f2b97b5fab" (UID: "5d76402c-4d39-4c03-8018-16f2b97b5fab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:45.342064 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.341860 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-tmp-dir" pod "a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" (UID: "a1d285f6-5118-41bc-aa3a-0c3348c8d1b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:45.342064 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.341902 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-home" pod "a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" (UID: "a1d285f6-5118-41bc-aa3a-0c3348c8d1b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:45.342064 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.341924 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-dshm\") pod \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " Nov 02 22:34:45.342064 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.341958 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d76402c-4d39-4c03-8018-16f2b97b5fab-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d76402c-4d39-4c03-8018-16f2b97b5fab-tls-certs\") pod \"5d76402c-4d39-4c03-8018-16f2b97b5fab\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " Nov 02 22:34:45.342064 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.342038 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-kube-api-access-kr2v4\" (UniqueName: \"kubernetes.io/projected/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-kube-api-access-kr2v4\") pod \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " Nov 02 22:34:45.342311 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.342085 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-home\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-home\") pod \"5d76402c-4d39-4c03-8018-16f2b97b5fab\" (UID: \"5d76402c-4d39-4c03-8018-16f2b97b5fab\") " Nov 02 22:34:45.342311 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.342131 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-tls-certs\") pod \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\" (UID: \"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6\") " Nov 02 22:34:45.342423 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.342390 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-home" pod "5d76402c-4d39-4c03-8018-16f2b97b5fab" (UID: "5d76402c-4d39-4c03-8018-16f2b97b5fab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:45.343040 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.342912 5871 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-home\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.343040 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.342942 5871 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-model-cache\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.343040 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.342960 5871 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-model-cache\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.343040 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.342976 5871 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-tmp-dir\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.343040 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.342993 5871 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-tmp-dir\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.343040 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.343007 5871 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-home\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.344445 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.344422 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-dshm" pod "5d76402c-4d39-4c03-8018-16f2b97b5fab" (UID: "5d76402c-4d39-4c03-8018-16f2b97b5fab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:45.344528 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.344452 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d76402c-4d39-4c03-8018-16f2b97b5fab-kube-api-access-pmk8w" pod "5d76402c-4d39-4c03-8018-16f2b97b5fab" (UID: "5d76402c-4d39-4c03-8018-16f2b97b5fab"). InnerVolumeSpecName "kube-api-access-pmk8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:45.344585 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.344522 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-dshm" pod "a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" (UID: "a1d285f6-5118-41bc-aa3a-0c3348c8d1b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:45.344585 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.344541 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-kube-api-access-kr2v4" pod "a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" (UID: "a1d285f6-5118-41bc-aa3a-0c3348c8d1b6"). InnerVolumeSpecName "kube-api-access-kr2v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:45.344657 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.344610 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d76402c-4d39-4c03-8018-16f2b97b5fab-tls-certs" pod "5d76402c-4d39-4c03-8018-16f2b97b5fab" (UID: "5d76402c-4d39-4c03-8018-16f2b97b5fab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:45.344690 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.344664 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-tls-certs" pod "a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" (UID: "a1d285f6-5118-41bc-aa3a-0c3348c8d1b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:34:45.443838 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.443790 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr2v4\" (UniqueName: \"kubernetes.io/projected/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-kube-api-access-kr2v4\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.443838 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.443835 5871 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-tls-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.444115 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.443856 5871 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d76402c-4d39-4c03-8018-16f2b97b5fab-dshm\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.444115 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.443889 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmk8w\" (UniqueName: \"kubernetes.io/projected/5d76402c-4d39-4c03-8018-16f2b97b5fab-kube-api-access-pmk8w\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.444115 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.443906 5871 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6-dshm\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.444115 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.443920 5871 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d76402c-4d39-4c03-8018-16f2b97b5fab-tls-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:45.580053 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.579965 5871 generic.go:356] "Generic (PLEG): container finished" podID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerID="c5859f1b20c931e286714f4815b82f7b86b1665c5cbd617df273cb8b5264c760" exitCode=0 Nov 02 22:34:45.580053 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.580014 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" Nov 02 22:34:45.580257 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.580047 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" event={"ID":"5d76402c-4d39-4c03-8018-16f2b97b5fab","Type":"ContainerDied","Data":"c5859f1b20c931e286714f4815b82f7b86b1665c5cbd617df273cb8b5264c760"} Nov 02 22:34:45.580257 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.580090 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz" event={"ID":"5d76402c-4d39-4c03-8018-16f2b97b5fab","Type":"ContainerDied","Data":"393ca4b6ae2a0accc4c04997036165e0bac0e2edb9cc52f24fee44d41d2da780"} Nov 02 22:34:45.580257 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.580113 5871 scope.go:122] "RemoveContainer" containerID="c5859f1b20c931e286714f4815b82f7b86b1665c5cbd617df273cb8b5264c760" Nov 02 22:34:45.581547 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.581519 5871 generic.go:356] "Generic (PLEG): container finished" podID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerID="7a9de685be59f2fbcedf0a63f9228f69e803cafea059231566802828ebe114d6" exitCode=0 Nov 02 22:34:45.581673 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.581557 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" Nov 02 22:34:45.581673 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.581596 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" event={"ID":"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6","Type":"ContainerDied","Data":"7a9de685be59f2fbcedf0a63f9228f69e803cafea059231566802828ebe114d6"} Nov 02 22:34:45.581673 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.581644 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p" event={"ID":"a1d285f6-5118-41bc-aa3a-0c3348c8d1b6","Type":"ContainerDied","Data":"50b1592c196264226e8f52c2b6ba261bafad275339c63948f0143d61fab57e26"} Nov 02 22:34:45.590816 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.590791 5871 scope.go:122] "RemoveContainer" containerID="c5859f1b20c931e286714f4815b82f7b86b1665c5cbd617df273cb8b5264c760" Nov 02 22:34:45.591186 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:34:45.591165 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5859f1b20c931e286714f4815b82f7b86b1665c5cbd617df273cb8b5264c760\": container with ID starting with c5859f1b20c931e286714f4815b82f7b86b1665c5cbd617df273cb8b5264c760 not found: ID does not exist" containerID="c5859f1b20c931e286714f4815b82f7b86b1665c5cbd617df273cb8b5264c760" Nov 02 22:34:45.591246 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.591197 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5859f1b20c931e286714f4815b82f7b86b1665c5cbd617df273cb8b5264c760"} err="failed to get container status \"c5859f1b20c931e286714f4815b82f7b86b1665c5cbd617df273cb8b5264c760\": rpc error: code = NotFound desc = could not find container \"c5859f1b20c931e286714f4815b82f7b86b1665c5cbd617df273cb8b5264c760\": container with ID starting with c5859f1b20c931e286714f4815b82f7b86b1665c5cbd617df273cb8b5264c760 not found: ID does not exist" Nov 02 22:34:45.591246 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.591223 5871 scope.go:122] "RemoveContainer" containerID="7a9de685be59f2fbcedf0a63f9228f69e803cafea059231566802828ebe114d6" Nov 02 22:34:45.600522 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.600493 5871 scope.go:122] "RemoveContainer" containerID="7a9de685be59f2fbcedf0a63f9228f69e803cafea059231566802828ebe114d6" Nov 02 22:34:45.600991 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:34:45.600905 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a9de685be59f2fbcedf0a63f9228f69e803cafea059231566802828ebe114d6\": container with ID starting with 7a9de685be59f2fbcedf0a63f9228f69e803cafea059231566802828ebe114d6 not found: ID does not exist" containerID="7a9de685be59f2fbcedf0a63f9228f69e803cafea059231566802828ebe114d6" Nov 02 22:34:45.600991 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.600942 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a9de685be59f2fbcedf0a63f9228f69e803cafea059231566802828ebe114d6"} err="failed to get container status \"7a9de685be59f2fbcedf0a63f9228f69e803cafea059231566802828ebe114d6\": rpc error: code = NotFound desc = could not find container \"7a9de685be59f2fbcedf0a63f9228f69e803cafea059231566802828ebe114d6\": container with ID starting with 7a9de685be59f2fbcedf0a63f9228f69e803cafea059231566802828ebe114d6 not found: ID does not exist" Nov 02 22:34:45.746933 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.746900 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p"] Nov 02 22:34:45.751161 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.751130 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8263a4/premium-w2-8263a4-kserve-7fd5b4d864-vsc2p"] Nov 02 22:34:45.860465 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.860380 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz"] Nov 02 22:34:45.863128 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:45.863107 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8263a4/distinct-w2-8263a4-kserve-94947676c-fblwz"] Nov 02 22:34:46.484784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:46.484727 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:47.312797 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:47.312765 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" path="/var/lib/kubelet/pods/5d76402c-4d39-4c03-8018-16f2b97b5fab/volumes" Nov 02 22:34:47.313230 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:47.313208 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" path="/var/lib/kubelet/pods/a1d285f6-5118-41bc-aa3a-0c3348c8d1b6/volumes" Nov 02 22:34:54.298769 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:54.298725 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" podUID="e449a219-b519-48b7-a4b0-b9dc589cdeb3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:34:57.509047 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:57.509007 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:59.032074 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:59.032033 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:34:59.048026 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:59.047994 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:59.316286 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:59.316209 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:34:59.963226 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:34:59.963188 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:02.274508 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:02.274438 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" Nov 02 22:35:02.375067 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:02.375032 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e449a219-b519-48b7-a4b0-b9dc589cdeb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e449a219-b519-48b7-a4b0-b9dc589cdeb3-plugins-config-volume\") pod \"e449a219-b519-48b7-a4b0-b9dc589cdeb3\" (UID: \"e449a219-b519-48b7-a4b0-b9dc589cdeb3\") " Nov 02 22:35:02.375249 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:02.375200 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e449a219-b519-48b7-a4b0-b9dc589cdeb3-kube-api-access-pk75v\" (UniqueName: \"kubernetes.io/projected/e449a219-b519-48b7-a4b0-b9dc589cdeb3-kube-api-access-pk75v\") pod \"e449a219-b519-48b7-a4b0-b9dc589cdeb3\" (UID: \"e449a219-b519-48b7-a4b0-b9dc589cdeb3\") " Nov 02 22:35:02.375514 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:02.375492 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e449a219-b519-48b7-a4b0-b9dc589cdeb3-plugins-config-volume" pod "e449a219-b519-48b7-a4b0-b9dc589cdeb3" (UID: "e449a219-b519-48b7-a4b0-b9dc589cdeb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:35:02.375609 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:02.375596 5871 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e449a219-b519-48b7-a4b0-b9dc589cdeb3-plugins-config-volume\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:02.377369 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:02.377309 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e449a219-b519-48b7-a4b0-b9dc589cdeb3-kube-api-access-pk75v" pod "e449a219-b519-48b7-a4b0-b9dc589cdeb3" (UID: "e449a219-b519-48b7-a4b0-b9dc589cdeb3"). InnerVolumeSpecName "kube-api-access-pk75v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:02.476791 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:02.476740 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk75v\" (UniqueName: \"kubernetes.io/projected/e449a219-b519-48b7-a4b0-b9dc589cdeb3-kube-api-access-pk75v\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:02.708943 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:02.708910 5871 generic.go:356] "Generic (PLEG): container finished" podID="e449a219-b519-48b7-a4b0-b9dc589cdeb3" containerID="856366419bb8331212bf34e7fd673fb0f2108883d28eda984a35687f02bdcf72" exitCode=0 Nov 02 22:35:02.709120 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:02.708962 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" Nov 02 22:35:02.709120 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:02.708964 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" event={"ID":"e449a219-b519-48b7-a4b0-b9dc589cdeb3","Type":"ContainerDied","Data":"856366419bb8331212bf34e7fd673fb0f2108883d28eda984a35687f02bdcf72"} Nov 02 22:35:02.709120 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:02.709090 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47" event={"ID":"e449a219-b519-48b7-a4b0-b9dc589cdeb3","Type":"ContainerDied","Data":"6c94f1945da0d311c9a3695deb92ecc2febeed707bc5da2c9ce0e3f2586e8a2b"} Nov 02 22:35:02.709230 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:02.709121 5871 scope.go:122] "RemoveContainer" containerID="856366419bb8331212bf34e7fd673fb0f2108883d28eda984a35687f02bdcf72" Nov 02 22:35:02.718767 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:02.718745 5871 scope.go:122] "RemoveContainer" containerID="856366419bb8331212bf34e7fd673fb0f2108883d28eda984a35687f02bdcf72" Nov 02 22:35:02.719195 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:35:02.719170 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856366419bb8331212bf34e7fd673fb0f2108883d28eda984a35687f02bdcf72\": container with ID starting with 856366419bb8331212bf34e7fd673fb0f2108883d28eda984a35687f02bdcf72 not found: ID does not exist" containerID="856366419bb8331212bf34e7fd673fb0f2108883d28eda984a35687f02bdcf72" Nov 02 22:35:02.719286 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:02.719210 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856366419bb8331212bf34e7fd673fb0f2108883d28eda984a35687f02bdcf72"} err="failed to get container status \"856366419bb8331212bf34e7fd673fb0f2108883d28eda984a35687f02bdcf72\": rpc error: code = NotFound desc = could not find container \"856366419bb8331212bf34e7fd673fb0f2108883d28eda984a35687f02bdcf72\": container with ID starting with 856366419bb8331212bf34e7fd673fb0f2108883d28eda984a35687f02bdcf72 not found: ID does not exist" Nov 02 22:35:02.875241 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:02.875199 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47"] Nov 02 22:35:02.925137 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:02.925091 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8263a4-7bdf7887b7-8sj47"] Nov 02 22:35:03.312387 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:03.312355 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e449a219-b519-48b7-a4b0-b9dc589cdeb3" path="/var/lib/kubelet/pods/e449a219-b519-48b7-a4b0-b9dc589cdeb3/volumes" Nov 02 22:35:06.360216 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:06.360180 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:07.538946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:07.538911 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c"] Nov 02 22:35:07.539388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:07.539129 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" podUID="1c46dd84-f664-4317-86e5-035c682cd8da" containerName="istio-proxy" containerID="cri-o://bd19281ed8bbd6a5c820345f2d3db518bc5d2636b29fc4084904eca820ea5680" gracePeriod=30 Nov 02 22:35:07.555209 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:07.555173 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p"] Nov 02 22:35:07.555398 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:07.555378 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" podUID="c30a7ffb-0fbd-437c-af03-a89ed7449bd4" containerName="istio-proxy" containerID="cri-o://6972d6132a0eed01e629f50a65098a05fc1f90068a427ff2e043fe85b199a2c3" gracePeriod=30 Nov 02 22:35:07.584087 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:07.584050 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7"] Nov 02 22:35:07.584323 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:07.584267 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" podUID="fae99bb6-0aee-4ef6-a938-f189add936ed" containerName="istio-proxy" containerID="cri-o://c2d217e5f1452decf2ef4f471490975ed9b146cb9873a2dc2e96be588acb917f" gracePeriod=30 Nov 02 22:35:08.214690 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:08.214646 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:08.734253 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:08.734214 5871 patch_prober.go:29] interesting pod/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:35:08.734755 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:08.734273 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" podUID="fae99bb6-0aee-4ef6-a938-f189add936ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:35:08.758530 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:08.758496 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:11.853107 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:11.853069 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:35:12.774151 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.773972 5871 generic.go:356] "Generic (PLEG): container finished" podID="c30a7ffb-0fbd-437c-af03-a89ed7449bd4" containerID="6972d6132a0eed01e629f50a65098a05fc1f90068a427ff2e043fe85b199a2c3" exitCode=0 Nov 02 22:35:12.774151 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.774092 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" event={"ID":"c30a7ffb-0fbd-437c-af03-a89ed7449bd4","Type":"ContainerDied","Data":"6972d6132a0eed01e629f50a65098a05fc1f90068a427ff2e043fe85b199a2c3"} Nov 02 22:35:12.776624 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.776583 5871 generic.go:356] "Generic (PLEG): container finished" podID="fae99bb6-0aee-4ef6-a938-f189add936ed" containerID="c2d217e5f1452decf2ef4f471490975ed9b146cb9873a2dc2e96be588acb917f" exitCode=0 Nov 02 22:35:12.776774 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.776646 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" event={"ID":"fae99bb6-0aee-4ef6-a938-f189add936ed","Type":"ContainerDied","Data":"c2d217e5f1452decf2ef4f471490975ed9b146cb9873a2dc2e96be588acb917f"} Nov 02 22:35:12.776774 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.776687 5871 scope.go:122] "RemoveContainer" containerID="aa30db04432008adadfc0c297ab3f6fce700209c999d2fec1716a46364a4e4b9" Nov 02 22:35:12.778904 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.778858 5871 generic.go:356] "Generic (PLEG): container finished" podID="1c46dd84-f664-4317-86e5-035c682cd8da" containerID="bd19281ed8bbd6a5c820345f2d3db518bc5d2636b29fc4084904eca820ea5680" exitCode=0 Nov 02 22:35:12.778973 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.778947 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" event={"ID":"1c46dd84-f664-4317-86e5-035c682cd8da","Type":"ContainerDied","Data":"bd19281ed8bbd6a5c820345f2d3db518bc5d2636b29fc4084904eca820ea5680"} Nov 02 22:35:12.848532 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.848505 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:35:12.886668 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.886641 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-istio-data\") pod \"1c46dd84-f664-4317-86e5-035c682cd8da\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " Nov 02 22:35:12.887098 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.886678 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-istio-envoy\") pod \"1c46dd84-f664-4317-86e5-035c682cd8da\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " Nov 02 22:35:12.887098 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.886711 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-workload-certs\") pod \"1c46dd84-f664-4317-86e5-035c682cd8da\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " Nov 02 22:35:12.887098 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.886777 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-credential-socket\") pod \"1c46dd84-f664-4317-86e5-035c682cd8da\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " Nov 02 22:35:12.887098 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.886822 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c46dd84-f664-4317-86e5-035c682cd8da-istio-token\" (UniqueName: \"kubernetes.io/projected/1c46dd84-f664-4317-86e5-035c682cd8da-istio-token\") pod \"1c46dd84-f664-4317-86e5-035c682cd8da\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " Nov 02 22:35:12.887098 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.886867 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c46dd84-f664-4317-86e5-035c682cd8da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c46dd84-f664-4317-86e5-035c682cd8da-istio-podinfo\") pod \"1c46dd84-f664-4317-86e5-035c682cd8da\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " Nov 02 22:35:12.887098 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.886971 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-workload-socket\") pod \"1c46dd84-f664-4317-86e5-035c682cd8da\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " Nov 02 22:35:12.887098 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.887019 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c46dd84-f664-4317-86e5-035c682cd8da-kube-api-access-l6tjf\" (UniqueName: \"kubernetes.io/projected/1c46dd84-f664-4317-86e5-035c682cd8da-kube-api-access-l6tjf\") pod \"1c46dd84-f664-4317-86e5-035c682cd8da\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " Nov 02 22:35:12.887098 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.887060 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-credential-socket" pod "1c46dd84-f664-4317-86e5-035c682cd8da" (UID: "1c46dd84-f664-4317-86e5-035c682cd8da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.887098 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.887075 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-istio-data" pod "1c46dd84-f664-4317-86e5-035c682cd8da" (UID: "1c46dd84-f664-4317-86e5-035c682cd8da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.887098 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.887072 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c46dd84-f664-4317-86e5-035c682cd8da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c46dd84-f664-4317-86e5-035c682cd8da-istiod-ca-cert\") pod \"1c46dd84-f664-4317-86e5-035c682cd8da\" (UID: \"1c46dd84-f664-4317-86e5-035c682cd8da\") " Nov 02 22:35:12.887098 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.887087 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-workload-certs" pod "1c46dd84-f664-4317-86e5-035c682cd8da" (UID: "1c46dd84-f664-4317-86e5-035c682cd8da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.887701 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.887297 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-workload-socket" pod "1c46dd84-f664-4317-86e5-035c682cd8da" (UID: "1c46dd84-f664-4317-86e5-035c682cd8da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.887701 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.887386 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c46dd84-f664-4317-86e5-035c682cd8da-istiod-ca-cert" pod "1c46dd84-f664-4317-86e5-035c682cd8da" (UID: "1c46dd84-f664-4317-86e5-035c682cd8da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:35:12.887701 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.887630 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c46dd84-f664-4317-86e5-035c682cd8da-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.887701 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.887650 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.887701 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.887663 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.887701 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.887678 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.887701 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.887692 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.888097 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.887734 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:35:12.889126 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.889101 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-istio-envoy" pod "1c46dd84-f664-4317-86e5-035c682cd8da" (UID: "1c46dd84-f664-4317-86e5-035c682cd8da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.889354 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.889335 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c46dd84-f664-4317-86e5-035c682cd8da-istio-token" pod "1c46dd84-f664-4317-86e5-035c682cd8da" (UID: "1c46dd84-f664-4317-86e5-035c682cd8da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:12.889503 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.889485 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c46dd84-f664-4317-86e5-035c682cd8da-kube-api-access-l6tjf" pod "1c46dd84-f664-4317-86e5-035c682cd8da" (UID: "1c46dd84-f664-4317-86e5-035c682cd8da"). InnerVolumeSpecName "kube-api-access-l6tjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:12.889589 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.889575 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c46dd84-f664-4317-86e5-035c682cd8da-istio-podinfo" pod "1c46dd84-f664-4317-86e5-035c682cd8da" (UID: "1c46dd84-f664-4317-86e5-035c682cd8da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:35:12.896886 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.896857 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:35:12.988556 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.988521 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-podinfo\") pod \"fae99bb6-0aee-4ef6-a938-f189add936ed\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " Nov 02 22:35:12.988556 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.988563 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-workload-certs\") pod \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " Nov 02 22:35:12.988953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.988596 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-data\") pod \"fae99bb6-0aee-4ef6-a938-f189add936ed\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " Nov 02 22:35:12.988953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.988632 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-envoy\") pod \"fae99bb6-0aee-4ef6-a938-f189add936ed\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " Nov 02 22:35:12.988953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.988658 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae99bb6-0aee-4ef6-a938-f189add936ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae99bb6-0aee-4ef6-a938-f189add936ed-istiod-ca-cert\") pod \"fae99bb6-0aee-4ef6-a938-f189add936ed\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " Nov 02 22:35:12.988953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.988684 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-envoy\") pod \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " Nov 02 22:35:12.988953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.988720 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-credential-socket\") pod \"fae99bb6-0aee-4ef6-a938-f189add936ed\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " Nov 02 22:35:12.988953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.988751 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istiod-ca-cert\") pod \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " Nov 02 22:35:12.988953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.988778 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-workload-socket\") pod \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " Nov 02 22:35:12.988953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.988816 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-credential-socket\") pod \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " Nov 02 22:35:12.988953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.988889 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-workload-socket\") pod \"fae99bb6-0aee-4ef6-a938-f189add936ed\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " Nov 02 22:35:12.988953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.988914 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-workload-certs" pod "c30a7ffb-0fbd-437c-af03-a89ed7449bd4" (UID: "c30a7ffb-0fbd-437c-af03-a89ed7449bd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.988953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.988936 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-token\" (UniqueName: \"kubernetes.io/projected/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-token\") pod \"fae99bb6-0aee-4ef6-a938-f189add936ed\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " Nov 02 22:35:12.989538 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.989005 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae99bb6-0aee-4ef6-a938-f189add936ed-kube-api-access-jx6h9\" (UniqueName: \"kubernetes.io/projected/fae99bb6-0aee-4ef6-a938-f189add936ed-kube-api-access-jx6h9\") pod \"fae99bb6-0aee-4ef6-a938-f189add936ed\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " Nov 02 22:35:12.989538 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.989059 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-token\" (UniqueName: \"kubernetes.io/projected/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-token\") pod \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " Nov 02 22:35:12.989538 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.989118 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-podinfo\") pod \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " Nov 02 22:35:12.989538 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.989159 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-data\") pod \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " Nov 02 22:35:12.989538 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.989197 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-workload-certs\") pod \"fae99bb6-0aee-4ef6-a938-f189add936ed\" (UID: \"fae99bb6-0aee-4ef6-a938-f189add936ed\") " Nov 02 22:35:12.989538 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.989278 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-kube-api-access-6qdz7\" (UniqueName: \"kubernetes.io/projected/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-kube-api-access-6qdz7\") pod \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\" (UID: \"c30a7ffb-0fbd-437c-af03-a89ed7449bd4\") " Nov 02 22:35:12.989538 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.989293 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-workload-socket" pod "c30a7ffb-0fbd-437c-af03-a89ed7449bd4" (UID: "c30a7ffb-0fbd-437c-af03-a89ed7449bd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.989538 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.989408 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-credential-socket" pod "c30a7ffb-0fbd-437c-af03-a89ed7449bd4" (UID: "c30a7ffb-0fbd-437c-af03-a89ed7449bd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.990358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.989733 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-credential-socket" pod "fae99bb6-0aee-4ef6-a938-f189add936ed" (UID: "fae99bb6-0aee-4ef6-a938-f189add936ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.990358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.989750 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-data" pod "c30a7ffb-0fbd-437c-af03-a89ed7449bd4" (UID: "c30a7ffb-0fbd-437c-af03-a89ed7449bd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.990358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.989781 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istiod-ca-cert" pod "c30a7ffb-0fbd-437c-af03-a89ed7449bd4" (UID: "c30a7ffb-0fbd-437c-af03-a89ed7449bd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:35:12.990358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.990045 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.990358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.990070 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6tjf\" (UniqueName: \"kubernetes.io/projected/1c46dd84-f664-4317-86e5-035c682cd8da-kube-api-access-l6tjf\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.990358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.990090 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.990358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.990105 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c46dd84-f664-4317-86e5-035c682cd8da-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.990358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.990120 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.990358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.990135 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c46dd84-f664-4317-86e5-035c682cd8da-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.990358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.990151 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c46dd84-f664-4317-86e5-035c682cd8da-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:12.990358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.990162 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae99bb6-0aee-4ef6-a938-f189add936ed-istiod-ca-cert" pod "fae99bb6-0aee-4ef6-a938-f189add936ed" (UID: "fae99bb6-0aee-4ef6-a938-f189add936ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:35:12.991783 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.991577 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-workload-certs" pod "fae99bb6-0aee-4ef6-a938-f189add936ed" (UID: "fae99bb6-0aee-4ef6-a938-f189add936ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.991783 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.991648 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-data" pod "fae99bb6-0aee-4ef6-a938-f189add936ed" (UID: "fae99bb6-0aee-4ef6-a938-f189add936ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.991783 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.991665 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-workload-socket" pod "fae99bb6-0aee-4ef6-a938-f189add936ed" (UID: "fae99bb6-0aee-4ef6-a938-f189add936ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.991783 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.991735 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-podinfo" pod "c30a7ffb-0fbd-437c-af03-a89ed7449bd4" (UID: "c30a7ffb-0fbd-437c-af03-a89ed7449bd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:35:12.991783 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.991745 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-envoy" pod "fae99bb6-0aee-4ef6-a938-f189add936ed" (UID: "fae99bb6-0aee-4ef6-a938-f189add936ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.991783 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.991750 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-envoy" pod "c30a7ffb-0fbd-437c-af03-a89ed7449bd4" (UID: "c30a7ffb-0fbd-437c-af03-a89ed7449bd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:35:12.992310 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.992266 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-token" pod "fae99bb6-0aee-4ef6-a938-f189add936ed" (UID: "fae99bb6-0aee-4ef6-a938-f189add936ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:12.992310 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.992301 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-kube-api-access-6qdz7" pod "c30a7ffb-0fbd-437c-af03-a89ed7449bd4" (UID: "c30a7ffb-0fbd-437c-af03-a89ed7449bd4"). InnerVolumeSpecName "kube-api-access-6qdz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:12.992555 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.992513 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-token" pod "c30a7ffb-0fbd-437c-af03-a89ed7449bd4" (UID: "c30a7ffb-0fbd-437c-af03-a89ed7449bd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:12.993330 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.993306 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae99bb6-0aee-4ef6-a938-f189add936ed-kube-api-access-jx6h9" pod "fae99bb6-0aee-4ef6-a938-f189add936ed" (UID: "fae99bb6-0aee-4ef6-a938-f189add936ed"). InnerVolumeSpecName "kube-api-access-jx6h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:12.993438 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:12.993422 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-podinfo" pod "fae99bb6-0aee-4ef6-a938-f189add936ed" (UID: "fae99bb6-0aee-4ef6-a938-f189add936ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:35:13.091623 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.091525 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:13.091623 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.091563 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:13.091623 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.091573 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:13.091623 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.091583 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:13.091623 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.091593 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qdz7\" (UniqueName: \"kubernetes.io/projected/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-kube-api-access-6qdz7\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:13.091623 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.091603 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:13.091623 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.091613 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:13.091623 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.091622 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:13.091623 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.091631 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae99bb6-0aee-4ef6-a938-f189add936ed-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:13.092081 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.091639 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:13.092081 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.091648 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:13.092081 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.091656 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c30a7ffb-0fbd-437c-af03-a89ed7449bd4-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:13.092081 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.091665 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae99bb6-0aee-4ef6-a938-f189add936ed-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:13.092081 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.091673 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae99bb6-0aee-4ef6-a938-f189add936ed-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:13.092081 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.091684 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx6h9\" (UniqueName: \"kubernetes.io/projected/fae99bb6-0aee-4ef6-a938-f189add936ed-kube-api-access-jx6h9\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:13.770421 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:35:13.770375 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:35:13.785608 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.785581 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" Nov 02 22:35:13.785794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.785604 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c" event={"ID":"1c46dd84-f664-4317-86e5-035c682cd8da","Type":"ContainerDied","Data":"204cc4a40b6a69fcfa09309661433c12fdaea3cb956f9486a15902484b0b69fb"} Nov 02 22:35:13.785794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.785697 5871 scope.go:122] "RemoveContainer" containerID="bd19281ed8bbd6a5c820345f2d3db518bc5d2636b29fc4084904eca820ea5680" Nov 02 22:35:13.787141 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.787114 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" event={"ID":"c30a7ffb-0fbd-437c-af03-a89ed7449bd4","Type":"ContainerDied","Data":"c60a9031536d5ed16033f0ffd5152f1cd6ffe2cc0f7956f8d4da22554f7ab927"} Nov 02 22:35:13.787218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.787210 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p" Nov 02 22:35:13.789448 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.789282 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" event={"ID":"fae99bb6-0aee-4ef6-a938-f189add936ed","Type":"ContainerDied","Data":"07cc786c9170f3a8f7cca19cfdec23f366b68f86a980b6962a06539c9071b16e"} Nov 02 22:35:13.789448 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.789334 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7" Nov 02 22:35:13.796326 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.796308 5871 scope.go:122] "RemoveContainer" containerID="6972d6132a0eed01e629f50a65098a05fc1f90068a427ff2e043fe85b199a2c3" Nov 02 22:35:13.805024 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.805002 5871 scope.go:122] "RemoveContainer" containerID="c2d217e5f1452decf2ef4f471490975ed9b146cb9873a2dc2e96be588acb917f" Nov 02 22:35:13.952229 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:13.952195 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7"] Nov 02 22:35:14.002334 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:14.002298 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-4v6g7"] Nov 02 22:35:14.070100 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:14.070055 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p"] Nov 02 22:35:14.074099 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:14.074066 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-qsx7p"] Nov 02 22:35:14.184827 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:14.184790 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c"] Nov 02 22:35:14.189095 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:14.189060 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8263a4-openshift-default-5bf6d64689-dc57c"] Nov 02 22:35:14.971966 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:14.971925 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:15.312992 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:15.312907 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c46dd84-f664-4317-86e5-035c682cd8da" path="/var/lib/kubelet/pods/1c46dd84-f664-4317-86e5-035c682cd8da/volumes" Nov 02 22:35:15.313363 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:15.313348 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c30a7ffb-0fbd-437c-af03-a89ed7449bd4" path="/var/lib/kubelet/pods/c30a7ffb-0fbd-437c-af03-a89ed7449bd4/volumes" Nov 02 22:35:15.313754 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:15.313740 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae99bb6-0aee-4ef6-a938-f189add936ed" path="/var/lib/kubelet/pods/fae99bb6-0aee-4ef6-a938-f189add936ed/volumes" Nov 02 22:35:43.000590 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:43.000554 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:43.769617 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:35:43.769576 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:35:46.826363 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:46.826310 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:56.215327 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:56.215291 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:35:58.836806 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:35:58.836757 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:07.905015 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:36:07.904972 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:13.769925 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:36:13.769883 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:36:15.484461 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:36:15.484413 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:43.423349 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:36:43.423312 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:36:43.769818 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:36:43.769733 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:36:45.667993 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:36:45.667956 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:36:47.206898 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:36:47.202403 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:37:07.065290 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:37:07.065251 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:37:08.259808 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:37:08.259755 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:37:13.832987 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:37:13.832918 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:37:17.646858 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:37:17.646814 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:37:38.554615 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:37:38.554573 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:37:43.712061 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:37:43.712026 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:37:43.770113 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:37:43.770073 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:37:47.555055 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:37:47.555020 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:05.328650 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:05.328570 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:38:08.719782 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:08.719742 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:13.769790 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:38:13.769751 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:38:35.018176 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:35.018140 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:35.201921 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:35.201861 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:36.030482 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:36.030448 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:38:42.748952 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:42.748910 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn"] Nov 02 22:38:42.749422 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:42.749114 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" podUID="8b20c5aa-5920-488b-a5ab-d22d40b7af57" containerName="discovery" containerID="cri-o://dc2826c458a2a0a6ca94f7a4e50090365fee6288ffccd542e2ca9b34c951c935" gracePeriod=30 Nov 02 22:38:43.003543 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.003475 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:38:43.061437 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.061399 5871 generic.go:356] "Generic (PLEG): container finished" podID="8b20c5aa-5920-488b-a5ab-d22d40b7af57" containerID="dc2826c458a2a0a6ca94f7a4e50090365fee6288ffccd542e2ca9b34c951c935" exitCode=0 Nov 02 22:38:43.061632 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.061446 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" Nov 02 22:38:43.061632 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.061481 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" event={"ID":"8b20c5aa-5920-488b-a5ab-d22d40b7af57","Type":"ContainerDied","Data":"dc2826c458a2a0a6ca94f7a4e50090365fee6288ffccd542e2ca9b34c951c935"} Nov 02 22:38:43.061632 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.061538 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn" event={"ID":"8b20c5aa-5920-488b-a5ab-d22d40b7af57","Type":"ContainerDied","Data":"992e822647b2441e2f67445150910eb0d72b5efccf72a0d386b11ec03ae513e7"} Nov 02 22:38:43.061632 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.061559 5871 scope.go:122] "RemoveContainer" containerID="dc2826c458a2a0a6ca94f7a4e50090365fee6288ffccd542e2ca9b34c951c935" Nov 02 22:38:43.071352 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.071329 5871 scope.go:122] "RemoveContainer" containerID="dc2826c458a2a0a6ca94f7a4e50090365fee6288ffccd542e2ca9b34c951c935" Nov 02 22:38:43.071634 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:38:43.071613 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc2826c458a2a0a6ca94f7a4e50090365fee6288ffccd542e2ca9b34c951c935\": container with ID starting with dc2826c458a2a0a6ca94f7a4e50090365fee6288ffccd542e2ca9b34c951c935 not found: ID does not exist" containerID="dc2826c458a2a0a6ca94f7a4e50090365fee6288ffccd542e2ca9b34c951c935" Nov 02 22:38:43.071682 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.071648 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc2826c458a2a0a6ca94f7a4e50090365fee6288ffccd542e2ca9b34c951c935"} err="failed to get container status \"dc2826c458a2a0a6ca94f7a4e50090365fee6288ffccd542e2ca9b34c951c935\": rpc error: code = NotFound desc = could not find container \"dc2826c458a2a0a6ca94f7a4e50090365fee6288ffccd542e2ca9b34c951c935\": container with ID starting with dc2826c458a2a0a6ca94f7a4e50090365fee6288ffccd542e2ca9b34c951c935 not found: ID does not exist" Nov 02 22:38:43.094010 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.093969 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-token\" (UniqueName: \"kubernetes.io/projected/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-token\") pod \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " Nov 02 22:38:43.094010 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.094013 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-csr-dns-cert\") pod \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " Nov 02 22:38:43.094296 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.094033 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b20c5aa-5920-488b-a5ab-d22d40b7af57-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b20c5aa-5920-488b-a5ab-d22d40b7af57-local-certs\") pod \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " Nov 02 22:38:43.094296 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.094081 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b20c5aa-5920-488b-a5ab-d22d40b7af57-kube-api-access-vxk28\" (UniqueName: \"kubernetes.io/projected/8b20c5aa-5920-488b-a5ab-d22d40b7af57-kube-api-access-vxk28\") pod \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " Nov 02 22:38:43.094296 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.094111 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-kubeconfig\") pod \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " Nov 02 22:38:43.094296 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.094138 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-csr-ca-configmap\") pod \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " Nov 02 22:38:43.094296 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.094253 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-cacerts\" (UniqueName: \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-cacerts\") pod \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\" (UID: \"8b20c5aa-5920-488b-a5ab-d22d40b7af57\") " Nov 02 22:38:43.094943 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.094911 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-csr-ca-configmap" pod "8b20c5aa-5920-488b-a5ab-d22d40b7af57" (UID: "8b20c5aa-5920-488b-a5ab-d22d40b7af57"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:38:43.096914 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.096739 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-kubeconfig" pod "8b20c5aa-5920-488b-a5ab-d22d40b7af57" (UID: "8b20c5aa-5920-488b-a5ab-d22d40b7af57"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:43.096914 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.096810 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-cacerts" pod "8b20c5aa-5920-488b-a5ab-d22d40b7af57" (UID: "8b20c5aa-5920-488b-a5ab-d22d40b7af57"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:43.096914 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.096815 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b20c5aa-5920-488b-a5ab-d22d40b7af57-local-certs" pod "8b20c5aa-5920-488b-a5ab-d22d40b7af57" (UID: "8b20c5aa-5920-488b-a5ab-d22d40b7af57"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:38:43.096914 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.096896 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b20c5aa-5920-488b-a5ab-d22d40b7af57-kube-api-access-vxk28" pod "8b20c5aa-5920-488b-a5ab-d22d40b7af57" (UID: "8b20c5aa-5920-488b-a5ab-d22d40b7af57"). InnerVolumeSpecName "kube-api-access-vxk28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:38:43.097119 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.096929 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-token" pod "8b20c5aa-5920-488b-a5ab-d22d40b7af57" (UID: "8b20c5aa-5920-488b-a5ab-d22d40b7af57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:38:43.097119 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.096945 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-csr-dns-cert" pod "8b20c5aa-5920-488b-a5ab-d22d40b7af57" (UID: "8b20c5aa-5920-488b-a5ab-d22d40b7af57"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:43.195410 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.195369 5871 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-cacerts\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:43.195410 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.195411 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:43.195625 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.195426 5871 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-csr-dns-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:43.195625 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.195441 5871 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b20c5aa-5920-488b-a5ab-d22d40b7af57-local-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:43.195625 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.195455 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxk28\" (UniqueName: \"kubernetes.io/projected/8b20c5aa-5920-488b-a5ab-d22d40b7af57-kube-api-access-vxk28\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:43.195625 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.195470 5871 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-kubeconfig\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:43.195625 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.195483 5871 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8b20c5aa-5920-488b-a5ab-d22d40b7af57-istio-csr-ca-configmap\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:43.477272 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.477223 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn"] Nov 02 22:38:43.481487 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:43.481453 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fnnvn"] Nov 02 22:38:43.770184 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:38:43.770148 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:38:44.071278 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:44.071191 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:45.046629 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:45.046579 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:45.231245 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:45.231203 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:45.313502 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:45.313419 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b20c5aa-5920-488b-a5ab-d22d40b7af57" path="/var/lib/kubelet/pods/8b20c5aa-5920-488b-a5ab-d22d40b7af57/volumes" Nov 02 22:38:46.746711 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:46.746669 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:38:47.899579 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:47.899544 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:50.607833 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:50.607767 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:58.259983 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:58.259924 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:38:59.090179 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:38:59.090140 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:39:01.753736 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:39:01.753698 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:01.928747 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:39:01.928708 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:08.695521 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:39:08.695484 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:13.770291 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:39:13.770252 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:39:25.250001 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:39:25.249908 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:39:34.649632 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:39:34.649589 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:39:43.769360 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:39:43.769322 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:39:44.431336 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:39:44.431302 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:39:53.364319 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:39:53.364278 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:40:05.748208 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:40:05.748170 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:40:13.770608 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:40:13.770570 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:40:14.068839 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:40:14.068753 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:22.384644 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:40:22.384609 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:22.561960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:40:22.561924 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:25.829361 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:40:25.829324 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:40:35.370346 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:40:35.370308 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:35.631839 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:40:35.631763 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:43.769056 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:40:43.769009 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:40:44.757373 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:40:44.757334 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:47.951763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:40:47.951724 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:40:53.249098 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:40:53.249058 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:57.486858 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:40:57.486766 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:40:57.665527 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:40:57.665495 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:01.028773 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:41:01.028734 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:41:09.831074 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:41:09.831018 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-pf98d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:41:09.831654 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:41:09.831107 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-pf98d" podUID="8fd7034d-1b03-47bb-bd55-53bbd67ec740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:41:11.781500 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:41:11.781464 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:41:13.769572 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:41:13.769536 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:41:14.773599 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:41:14.773562 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:14.999604 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:41:14.999577 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:31.731309 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:41:31.731271 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:41:41.344364 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:41:41.344330 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:41:43.769553 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:41:43.769519 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:41:45.168712 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:41:45.168673 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:53.573935 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:41:53.573858 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:41:57.465708 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:41:57.465668 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:42:13.770815 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:42:13.770776 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:42:19.421183 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:42:19.421139 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:30.765514 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:42:30.765428 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:42:39.870001 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:42:39.869959 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:40.050592 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:42:40.050555 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:41.464638 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:42:41.464600 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:42:43.770351 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:42:43.770313 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:42:45.439531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:42:45.439491 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:42:53.918868 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:42:53.918831 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:10.731330 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:10.731289 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:43:13.771058 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:43:13.771015 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:43:19.781919 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:19.781864 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:20.866578 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:20.866537 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:43:28.660903 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:28.660850 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:29.170081 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170038 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2"] Nov 02 22:43:29.170518 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170499 5871 cpu_manager.go:414] "containerMap: removing container" podUID="4824cb6c-15d6-4c48-8824-d7bf7b7932f1" containerName="istio-proxy" Nov 02 22:43:29.170615 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170520 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4824cb6c-15d6-4c48-8824-d7bf7b7932f1" containerName="istio-proxy" Nov 02 22:43:29.170615 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170543 5871 cpu_manager.go:414] "containerMap: removing container" podUID="291b0b67-ee70-4d16-ac08-8064e7f0fef1" containerName="istio-proxy" Nov 02 22:43:29.170615 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170551 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="291b0b67-ee70-4d16-ac08-8064e7f0fef1" containerName="istio-proxy" Nov 02 22:43:29.170615 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170565 5871 cpu_manager.go:414] "containerMap: removing container" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" Nov 02 22:43:29.170615 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170573 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" Nov 02 22:43:29.170615 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170586 5871 cpu_manager.go:414] "containerMap: removing container" podUID="20f6517a-1f4a-4eb9-b7cc-4b02d76cacae" containerName="maas-api" Nov 02 22:43:29.170615 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170594 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f6517a-1f4a-4eb9-b7cc-4b02d76cacae" containerName="maas-api" Nov 02 22:43:29.170615 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170604 5871 cpu_manager.go:414] "containerMap: removing container" podUID="8b20c5aa-5920-488b-a5ab-d22d40b7af57" containerName="discovery" Nov 02 22:43:29.170615 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170616 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b20c5aa-5920-488b-a5ab-d22d40b7af57" containerName="discovery" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170634 5871 cpu_manager.go:414] "containerMap: removing container" podUID="fae99bb6-0aee-4ef6-a938-f189add936ed" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170641 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae99bb6-0aee-4ef6-a938-f189add936ed" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170652 5871 cpu_manager.go:414] "containerMap: removing container" podUID="23563615-f0f4-4cb4-9159-815000b8fa24" containerName="maas-api" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170659 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23563615-f0f4-4cb4-9159-815000b8fa24" containerName="maas-api" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170670 5871 cpu_manager.go:414] "containerMap: removing container" podUID="3aa92502-a30d-4230-9338-80d73e06c66f" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170679 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa92502-a30d-4230-9338-80d73e06c66f" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170691 5871 cpu_manager.go:414] "containerMap: removing container" podUID="b03723b8-b2e9-4f09-9054-c069f1f5f45c" containerName="maas-api" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170698 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b03723b8-b2e9-4f09-9054-c069f1f5f45c" containerName="maas-api" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170709 5871 cpu_manager.go:414] "containerMap: removing container" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170716 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170728 5871 cpu_manager.go:414] "containerMap: removing container" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170734 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170749 5871 cpu_manager.go:414] "containerMap: removing container" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170756 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170767 5871 cpu_manager.go:414] "containerMap: removing container" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170774 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170783 5871 cpu_manager.go:414] "containerMap: removing container" podUID="9384cb9b-0f07-485e-9635-c7dc585a8b76" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170791 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9384cb9b-0f07-485e-9635-c7dc585a8b76" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170803 5871 cpu_manager.go:414] "containerMap: removing container" podUID="3782bc26-cb56-41c1-b1d2-608d185145fc" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170810 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3782bc26-cb56-41c1-b1d2-608d185145fc" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170822 5871 cpu_manager.go:414] "containerMap: removing container" podUID="1c46dd84-f664-4317-86e5-035c682cd8da" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170829 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c46dd84-f664-4317-86e5-035c682cd8da" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170839 5871 cpu_manager.go:414] "containerMap: removing container" podUID="5c65b13b-f4dd-4a8a-9844-a0f0309be485" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170846 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c65b13b-f4dd-4a8a-9844-a0f0309be485" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170857 5871 cpu_manager.go:414] "containerMap: removing container" podUID="01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170863 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170902 5871 cpu_manager.go:414] "containerMap: removing container" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170909 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170921 5871 cpu_manager.go:414] "containerMap: removing container" podUID="4839ef24-050d-41ea-95c3-7713d6df39a0" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170928 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4839ef24-050d-41ea-95c3-7713d6df39a0" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170937 5871 cpu_manager.go:414] "containerMap: removing container" podUID="667069a7-93cf-42ef-92ec-e825e6f6fabd" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170945 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="667069a7-93cf-42ef-92ec-e825e6f6fabd" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170963 5871 cpu_manager.go:414] "containerMap: removing container" podUID="e449a219-b519-48b7-a4b0-b9dc589cdeb3" containerName="payload-pre-processing" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170973 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e449a219-b519-48b7-a4b0-b9dc589cdeb3" containerName="payload-pre-processing" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170984 5871 cpu_manager.go:414] "containerMap: removing container" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.170991 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171003 5871 cpu_manager.go:414] "containerMap: removing container" podUID="e364f2d9-c537-4289-b605-547836336c31" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171011 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e364f2d9-c537-4289-b605-547836336c31" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171025 5871 cpu_manager.go:414] "containerMap: removing container" podUID="4839ef24-050d-41ea-95c3-7713d6df39a0" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171032 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4839ef24-050d-41ea-95c3-7713d6df39a0" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171041 5871 cpu_manager.go:414] "containerMap: removing container" podUID="fd88af66-9693-4cd1-abd5-d2af89caa020" containerName="maas-api" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171049 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd88af66-9693-4cd1-abd5-d2af89caa020" containerName="maas-api" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171058 5871 cpu_manager.go:414] "containerMap: removing container" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171066 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171080 5871 cpu_manager.go:414] "containerMap: removing container" podUID="7c7a6cfc-826c-4c21-8d77-35272c34d117" containerName="maas-api" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171087 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7a6cfc-826c-4c21-8d77-35272c34d117" containerName="maas-api" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171099 5871 cpu_manager.go:414] "containerMap: removing container" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171105 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171119 5871 cpu_manager.go:414] "containerMap: removing container" podUID="c30a7ffb-0fbd-437c-af03-a89ed7449bd4" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171126 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c30a7ffb-0fbd-437c-af03-a89ed7449bd4" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171136 5871 cpu_manager.go:414] "containerMap: removing container" podUID="2278f047-8cff-4d4f-b799-5b50740987c8" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171143 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2278f047-8cff-4d4f-b799-5b50740987c8" containerName="istio-proxy" Nov 02 22:43:29.171130 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171154 5871 cpu_manager.go:414] "containerMap: removing container" podUID="dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171161 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171172 5871 cpu_manager.go:414] "containerMap: removing container" podUID="fae99bb6-0aee-4ef6-a938-f189add936ed" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171179 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae99bb6-0aee-4ef6-a938-f189add936ed" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171188 5871 cpu_manager.go:414] "containerMap: removing container" podUID="8c608b09-9551-4d42-91c6-f6cb1a38bab7" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171196 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c608b09-9551-4d42-91c6-f6cb1a38bab7" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171208 5871 cpu_manager.go:414] "containerMap: removing container" podUID="3210ef32-d6e6-47d3-b431-826aba457e12" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171215 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3210ef32-d6e6-47d3-b431-826aba457e12" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171225 5871 cpu_manager.go:414] "containerMap: removing container" podUID="26990ef1-f003-4196-b347-9dbf761a25ec" containerName="maas-api" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171232 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26990ef1-f003-4196-b347-9dbf761a25ec" containerName="maas-api" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171247 5871 cpu_manager.go:414] "containerMap: removing container" podUID="f2d38074-6059-4106-96a0-46fff0b1663c" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171254 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2d38074-6059-4106-96a0-46fff0b1663c" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171265 5871 cpu_manager.go:414] "containerMap: removing container" podUID="16b4519b-9df1-4c63-9f22-d46e804efef4" containerName="maas-api" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171272 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b4519b-9df1-4c63-9f22-d46e804efef4" containerName="maas-api" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171390 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="291b0b67-ee70-4d16-ac08-8064e7f0fef1" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171401 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b4519b-9df1-4c63-9f22-d46e804efef4" containerName="maas-api" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171410 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2d38074-6059-4106-96a0-46fff0b1663c" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171424 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="4824cb6c-15d6-4c48-8824-d7bf7b7932f1" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171435 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="291b0b67-ee70-4d16-ac08-8064e7f0fef1" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171445 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d76402c-4d39-4c03-8018-16f2b97b5fab" containerName="main" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171455 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f6517a-1f4a-4eb9-b7cc-4b02d76cacae" containerName="maas-api" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171463 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b20c5aa-5920-488b-a5ab-d22d40b7af57" containerName="discovery" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171474 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae99bb6-0aee-4ef6-a938-f189add936ed" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171483 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="23563615-f0f4-4cb4-9159-815000b8fa24" containerName="maas-api" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171491 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa92502-a30d-4230-9338-80d73e06c66f" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171501 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="b03723b8-b2e9-4f09-9054-c069f1f5f45c" containerName="maas-api" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171508 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="b70d1226-ba5d-4cf0-af76-4852ce82bd76" containerName="main" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171517 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c1aa58-b96e-4539-b445-62d8cb67ffa9" containerName="main" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171526 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f259989-289c-4b0d-a158-9755a2ecb9fe" containerName="main" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171535 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="d705b15d-5840-41da-b5d8-aad314aeb33b" containerName="main" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171545 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="9384cb9b-0f07-485e-9635-c7dc585a8b76" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171552 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="3782bc26-cb56-41c1-b1d2-608d185145fc" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171562 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c46dd84-f664-4317-86e5-035c682cd8da" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171570 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c65b13b-f4dd-4a8a-9844-a0f0309be485" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171581 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="01dbb32b-088f-4cb3-8c0e-c845fcefc0ce" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171589 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="36cf5632-ebba-49b5-ad2c-40f5004570bd" containerName="main" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171598 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="4839ef24-050d-41ea-95c3-7713d6df39a0" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171606 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="667069a7-93cf-42ef-92ec-e825e6f6fabd" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171618 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="e449a219-b519-48b7-a4b0-b9dc589cdeb3" containerName="payload-pre-processing" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171627 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="180c8830-c2d7-42b4-9a5e-41f15996a8d6" containerName="main" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171636 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="e364f2d9-c537-4289-b605-547836336c31" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171647 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd88af66-9693-4cd1-abd5-d2af89caa020" containerName="maas-api" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171655 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c57bc80-ff20-4fc4-a19d-5dce6dc11ede" containerName="main" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171663 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7a6cfc-826c-4c21-8d77-35272c34d117" containerName="maas-api" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171673 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d285f6-5118-41bc-aa3a-0c3348c8d1b6" containerName="main" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171681 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="c30a7ffb-0fbd-437c-af03-a89ed7449bd4" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171688 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="2278f047-8cff-4d4f-b799-5b50740987c8" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171698 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd8e7b9c-7a3a-4c87-b8b9-ab242f263124" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171706 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae99bb6-0aee-4ef6-a938-f189add936ed" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171715 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c608b09-9551-4d42-91c6-f6cb1a38bab7" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171724 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="3210ef32-d6e6-47d3-b431-826aba457e12" containerName="istio-proxy" Nov 02 22:43:29.173455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.171732 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="26990ef1-f003-4196-b347-9dbf761a25ec" containerName="maas-api" Nov 02 22:43:29.175847 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.175826 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.228791 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.228760 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-a8baae-openshift-default-dockercfg-fp2kp\"" Nov 02 22:43:29.280891 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.280845 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2"] Nov 02 22:43:29.282650 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.282628 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-credential-socket\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.282721 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.282660 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-workload-certs\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.282797 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.282776 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knb6t\" (UniqueName: \"kubernetes.io/projected/2937a840-e872-473e-a2c7-777873cde2fb-kube-api-access-knb6t\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.282855 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.282823 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-workload-socket\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.282933 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.282891 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-istio-envoy\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.282996 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.282942 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-istio-data\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.283054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.282999 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2937a840-e872-473e-a2c7-777873cde2fb-istiod-ca-cert\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.283054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.283040 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2937a840-e872-473e-a2c7-777873cde2fb-istio-podinfo\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.283146 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.283068 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2937a840-e872-473e-a2c7-777873cde2fb-istio-token\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.295475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.295446 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:29.384520 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.384476 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2937a840-e872-473e-a2c7-777873cde2fb-istio-podinfo\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.384520 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.384525 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2937a840-e872-473e-a2c7-777873cde2fb-istio-token\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.384790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.384605 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-credential-socket\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.384790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.384630 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-workload-certs\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.384790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.384696 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knb6t\" (UniqueName: \"kubernetes.io/projected/2937a840-e872-473e-a2c7-777873cde2fb-kube-api-access-knb6t\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.384790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.384724 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-workload-socket\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.384790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.384758 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-istio-envoy\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.385102 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.384827 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-istio-data\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.385102 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.384859 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2937a840-e872-473e-a2c7-777873cde2fb-istiod-ca-cert\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.385216 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.385105 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-credential-socket\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.385216 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.385158 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-istio-data\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.385216 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.385205 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-workload-socket\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.385401 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.385381 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-workload-certs\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.385628 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.385607 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2937a840-e872-473e-a2c7-777873cde2fb-istiod-ca-cert\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.387057 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.387035 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2937a840-e872-473e-a2c7-777873cde2fb-istio-podinfo\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.387145 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.387040 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-istio-envoy\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.445934 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.445840 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2937a840-e872-473e-a2c7-777873cde2fb-istio-token\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.446098 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.446028 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knb6t\" (UniqueName: \"kubernetes.io/projected/2937a840-e872-473e-a2c7-777873cde2fb-kube-api-access-knb6t\") pod \"e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.477287 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.477240 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:29.488931 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.488899 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:29.921558 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.921532 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2"] Nov 02 22:43:29.923945 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:43:29.923914 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2937a840_e872_473e_a2c7_777873cde2fb.slice/crio-2cdc1214f21383b1e1b98caa5c00b4dd917b105e4757e6265d54ecd0807d9d62 WatchSource:0}: Error finding container 2cdc1214f21383b1e1b98caa5c00b4dd917b105e4757e6265d54ecd0807d9d62: Status 404 returned error can't find the container with id 2cdc1214f21383b1e1b98caa5c00b4dd917b105e4757e6265d54ecd0807d9d62 Nov 02 22:43:29.926242 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.926209 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:43:29.926311 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.926281 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:43:29.926354 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:29.926308 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:43:30.796568 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:30.796532 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" event={"ID":"2937a840-e872-473e-a2c7-777873cde2fb","Type":"ContainerStarted","Data":"6182aff009a0dc479ae5f3472f8f4bef2a5fc8d884b9e5af97e499332c4e74f0"} Nov 02 22:43:30.796792 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:30.796591 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" event={"ID":"2937a840-e872-473e-a2c7-777873cde2fb","Type":"ContainerStarted","Data":"2cdc1214f21383b1e1b98caa5c00b4dd917b105e4757e6265d54ecd0807d9d62"} Nov 02 22:43:30.968048 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:30.967989 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" podStartSLOduration=1.967975545 podStartE2EDuration="1.967975545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:43:30.966615478 +0000 UTC m=+2233.778945655" watchObservedRunningTime="2026-11-02 22:43:30.967975545 +0000 UTC m=+2233.780305722" Nov 02 22:43:31.489494 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:31.489452 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:31.494761 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:31.494732 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:31.802055 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:31.801972 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:31.803101 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:31.803085 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:33.935656 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:33.935598 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:34.118400 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:34.118365 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:34.290332 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:34.290222 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:40.413918 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.413863 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9"] Nov 02 22:43:40.414342 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.414325 5871 cpu_manager.go:414] "containerMap: removing container" podUID="291b0b67-ee70-4d16-ac08-8064e7f0fef1" containerName="istio-proxy" Nov 02 22:43:40.414387 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.414343 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="291b0b67-ee70-4d16-ac08-8064e7f0fef1" containerName="istio-proxy" Nov 02 22:43:40.414447 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.414438 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="4839ef24-050d-41ea-95c3-7713d6df39a0" containerName="istio-proxy" Nov 02 22:43:40.417485 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.417467 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:40.470438 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.470403 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-a8baae-serving-cert\"" Nov 02 22:43:40.500451 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.500413 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e51b93-dd50-485c-a163-6bc7387c9392-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:40.500635 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.500483 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8ff7\" (UniqueName: \"kubernetes.io/projected/01e51b93-dd50-485c-a163-6bc7387c9392-kube-api-access-q8ff7\") pod \"maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:40.500635 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.500563 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:40.500753 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.500729 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:40.523142 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.523097 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9"] Nov 02 22:43:40.601968 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.601920 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:40.602177 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.602038 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e51b93-dd50-485c-a163-6bc7387c9392-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:40.602177 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.602077 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8ff7\" (UniqueName: \"kubernetes.io/projected/01e51b93-dd50-485c-a163-6bc7387c9392-kube-api-access-q8ff7\") pod \"maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:40.602177 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.602099 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:40.602355 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:43:40.602257 5871 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-serving-cert: secret "maas-api-e2e-worker-single-main-a8baae-serving-cert" not found Nov 02 22:43:40.602355 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:43:40.602326 5871 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-maas-api-tls podName:01e51b93-dd50-485c-a163-6bc7387c9392 nodeName:}" failed. No retries permitted until 2026-11-02 22:43:41.102302513 +0000 UTC m=+2243.914632686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-maas-api-tls") pod "maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" (UID: "01e51b93-dd50-485c-a163-6bc7387c9392") : secret "maas-api-e2e-worker-single-main-a8baae-serving-cert" not found Nov 02 22:43:40.602766 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.602743 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e51b93-dd50-485c-a163-6bc7387c9392-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:40.604450 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.604428 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:40.641415 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.641361 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:40.666938 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.666814 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8ff7\" (UniqueName: \"kubernetes.io/projected/01e51b93-dd50-485c-a163-6bc7387c9392-kube-api-access-q8ff7\") pod \"maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:40.822526 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.822475 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:40.991441 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:40.991353 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:41.107269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:41.107229 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:41.109789 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:41.109767 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:41.332005 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:41.331908 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:41.710354 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:41.710314 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9"] Nov 02 22:43:41.714164 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:43:41.714127 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e51b93_dd50_485c_a163_6bc7387c9392.slice/crio-76c5d4363542a415e1e4e532f42de15be72c8a99ea1328f3e0087c28f932a418 WatchSource:0}: Error finding container 76c5d4363542a415e1e4e532f42de15be72c8a99ea1328f3e0087c28f932a418: Status 404 returned error can't find the container with id 76c5d4363542a415e1e4e532f42de15be72c8a99ea1328f3e0087c28f932a418 Nov 02 22:43:41.715690 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:41.715665 5871 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:43:41.857797 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:41.857754 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" event={"ID":"01e51b93-dd50-485c-a163-6bc7387c9392","Type":"ContainerStarted","Data":"76c5d4363542a415e1e4e532f42de15be72c8a99ea1328f3e0087c28f932a418"} Nov 02 22:43:42.865391 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:42.865347 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" event={"ID":"01e51b93-dd50-485c-a163-6bc7387c9392","Type":"ContainerStarted","Data":"ec6ab8687542e9f54c4123790e5904367536a0fbbee73698c40185c59e3487ed"} Nov 02 22:43:42.865832 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:42.865493 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:43.002957 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:43.002827 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" podStartSLOduration=2.005420946 podStartE2EDuration="3.002810591s" totalImagesPullingTime="997.389645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:43:43.000511225 +0000 UTC m=+2245.812841423" watchObservedRunningTime="2026-11-02 22:43:43.002810591 +0000 UTC m=+2245.815140768" Nov 02 22:43:43.770806 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:43:43.770766 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:43:45.702706 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:45.702666 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:45.877477 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:45.877437 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:46.268538 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:46.268500 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:46.445617 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:46.445586 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:48.331731 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:48.331689 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:43:48.881706 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:48.881674 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:43:50.792067 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:50.792032 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:54.249470 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:54.249413 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:55.701612 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:55.701577 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:57.959012 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:57.958973 5871 generic.go:356] "Generic (PLEG): container finished" podID="2937a840-e872-473e-a2c7-777873cde2fb" containerID="6182aff009a0dc479ae5f3472f8f4bef2a5fc8d884b9e5af97e499332c4e74f0" exitCode=0 Nov 02 22:43:57.959522 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:57.959050 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" event={"ID":"2937a840-e872-473e-a2c7-777873cde2fb","Type":"ContainerDied","Data":"6182aff009a0dc479ae5f3472f8f4bef2a5fc8d884b9e5af97e499332c4e74f0"} Nov 02 22:43:57.959522 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:57.959493 5871 scope.go:122] "RemoveContainer" containerID="6182aff009a0dc479ae5f3472f8f4bef2a5fc8d884b9e5af97e499332c4e74f0" Nov 02 22:43:57.960784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:57.960758 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:43:57.960839 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:57.960826 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:43:57.960889 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:57.960861 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:43:58.686487 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:58.686455 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:58.983144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:58.983047 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" event={"ID":"2937a840-e872-473e-a2c7-777873cde2fb","Type":"ContainerStarted","Data":"616b7d1ffefbae322a07edfd0e992acdcb66c323a84d789321e619f3455a247e"} Nov 02 22:43:59.280838 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:59.280744 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:59.484917 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:59.484858 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:43:59.489581 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:59.489550 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:43:59.490550 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:59.490523 5871 patch_prober.go:29] interesting pod/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:43:59.490690 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:43:59.490590 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" podUID="2937a840-e872-473e-a2c7-777873cde2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 22:44:00.489666 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:00.489630 5871 patch_prober.go:29] interesting pod/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:44:00.490193 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:00.489691 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" podUID="2937a840-e872-473e-a2c7-777873cde2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 22:44:01.107246 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:01.107206 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:01.489977 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:01.489941 5871 patch_prober.go:29] interesting pod/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:44:01.490379 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:01.489999 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" podUID="2937a840-e872-473e-a2c7-777873cde2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 22:44:01.802266 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:01.802170 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:44:02.418205 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:02.418164 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:02.489910 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:02.489858 5871 patch_prober.go:29] interesting pod/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:44:02.490093 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:02.489927 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" podUID="2937a840-e872-473e-a2c7-777873cde2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 22:44:03.489491 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:03.489450 5871 patch_prober.go:29] interesting pod/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 22:44:03.489689 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:03.489509 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" podUID="2937a840-e872-473e-a2c7-777873cde2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 22:44:04.494198 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:04.494161 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:44:04.495334 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:04.495314 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:44:06.086721 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:06.086684 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:06.779738 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:06.779703 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:44:09.091400 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:09.091358 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:11.090453 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:11.090413 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:13.770743 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:44:13.770698 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:44:16.008574 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:16.008538 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:16.190913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:16.190868 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:20.108371 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:20.108331 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:20.990114 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:20.990077 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:25.918497 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:25.918450 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:30.911327 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:30.911251 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:32.370470 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:32.370428 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:44:35.565540 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:35.565503 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:35.906069 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:35.906037 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:36.500989 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:36.500956 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:40.878846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:40.878808 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:43.770665 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:44:43.770619 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:44:45.789885 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:45.789845 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:50.803799 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:50.803765 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:54.586992 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:54.586956 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:55.702717 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:55.702677 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:56.149675 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:56.149637 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:44:58.049084 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:44:58.049043 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:00.892495 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:00.892461 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:01.027475 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:01.027431 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:45:03.077680 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:03.077644 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:06.090097 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:06.090059 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:10.321540 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:10.321500 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9"] Nov 02 22:45:10.322074 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:10.321743 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" podUID="01e51b93-dd50-485c-a163-6bc7387c9392" containerName="maas-api" containerID="cri-o://ec6ab8687542e9f54c4123790e5904367536a0fbbee73698c40185c59e3487ed" gracePeriod=30 Nov 02 22:45:10.451704 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:10.451671 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:10.625344 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:10.625311 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:11.575588 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:11.575558 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:45:11.642787 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:11.642752 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-maas-api-tls\") pod \"01e51b93-dd50-485c-a163-6bc7387c9392\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " Nov 02 22:45:11.643006 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:11.642997 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e51b93-dd50-485c-a163-6bc7387c9392-kube-api-access-q8ff7\" (UniqueName: \"kubernetes.io/projected/01e51b93-dd50-485c-a163-6bc7387c9392-kube-api-access-q8ff7\") pod \"01e51b93-dd50-485c-a163-6bc7387c9392\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " Nov 02 22:45:11.643054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:11.643034 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-metrics-certs\") pod \"01e51b93-dd50-485c-a163-6bc7387c9392\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " Nov 02 22:45:11.643104 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:11.643089 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e51b93-dd50-485c-a163-6bc7387c9392-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e51b93-dd50-485c-a163-6bc7387c9392-openshift-service-ca\") pod \"01e51b93-dd50-485c-a163-6bc7387c9392\" (UID: \"01e51b93-dd50-485c-a163-6bc7387c9392\") " Nov 02 22:45:11.643531 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:11.643499 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e51b93-dd50-485c-a163-6bc7387c9392-openshift-service-ca" pod "01e51b93-dd50-485c-a163-6bc7387c9392" (UID: "01e51b93-dd50-485c-a163-6bc7387c9392"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:45:11.644976 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:11.644949 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-maas-api-tls" pod "01e51b93-dd50-485c-a163-6bc7387c9392" (UID: "01e51b93-dd50-485c-a163-6bc7387c9392"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:45:11.645147 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:11.645123 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-metrics-certs" pod "01e51b93-dd50-485c-a163-6bc7387c9392" (UID: "01e51b93-dd50-485c-a163-6bc7387c9392"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:45:11.645197 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:11.645165 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e51b93-dd50-485c-a163-6bc7387c9392-kube-api-access-q8ff7" pod "01e51b93-dd50-485c-a163-6bc7387c9392" (UID: "01e51b93-dd50-485c-a163-6bc7387c9392"). InnerVolumeSpecName "kube-api-access-q8ff7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:45:11.744577 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:11.744487 5871 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e51b93-dd50-485c-a163-6bc7387c9392-openshift-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:11.744577 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:11.744520 5871 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-maas-api-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:11.744577 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:11.744533 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8ff7\" (UniqueName: \"kubernetes.io/projected/01e51b93-dd50-485c-a163-6bc7387c9392-kube-api-access-q8ff7\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:11.744577 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:11.744545 5871 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01e51b93-dd50-485c-a163-6bc7387c9392-metrics-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:12.387039 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:12.387001 5871 generic.go:356] "Generic (PLEG): container finished" podID="01e51b93-dd50-485c-a163-6bc7387c9392" containerID="ec6ab8687542e9f54c4123790e5904367536a0fbbee73698c40185c59e3487ed" exitCode=0 Nov 02 22:45:12.387246 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:12.387048 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" Nov 02 22:45:12.387246 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:12.387079 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" event={"ID":"01e51b93-dd50-485c-a163-6bc7387c9392","Type":"ContainerDied","Data":"ec6ab8687542e9f54c4123790e5904367536a0fbbee73698c40185c59e3487ed"} Nov 02 22:45:12.387246 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:12.387140 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9" event={"ID":"01e51b93-dd50-485c-a163-6bc7387c9392","Type":"ContainerDied","Data":"76c5d4363542a415e1e4e532f42de15be72c8a99ea1328f3e0087c28f932a418"} Nov 02 22:45:12.387246 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:12.387168 5871 scope.go:122] "RemoveContainer" containerID="ec6ab8687542e9f54c4123790e5904367536a0fbbee73698c40185c59e3487ed" Nov 02 22:45:12.397546 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:12.397528 5871 scope.go:122] "RemoveContainer" containerID="ec6ab8687542e9f54c4123790e5904367536a0fbbee73698c40185c59e3487ed" Nov 02 22:45:12.397840 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:45:12.397820 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6ab8687542e9f54c4123790e5904367536a0fbbee73698c40185c59e3487ed\": container with ID starting with ec6ab8687542e9f54c4123790e5904367536a0fbbee73698c40185c59e3487ed not found: ID does not exist" containerID="ec6ab8687542e9f54c4123790e5904367536a0fbbee73698c40185c59e3487ed" Nov 02 22:45:12.397923 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:12.397850 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6ab8687542e9f54c4123790e5904367536a0fbbee73698c40185c59e3487ed"} err="failed to get container status \"ec6ab8687542e9f54c4123790e5904367536a0fbbee73698c40185c59e3487ed\": rpc error: code = NotFound desc = could not find container \"ec6ab8687542e9f54c4123790e5904367536a0fbbee73698c40185c59e3487ed\": container with ID starting with ec6ab8687542e9f54c4123790e5904367536a0fbbee73698c40185c59e3487ed not found: ID does not exist" Nov 02 22:45:12.586810 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:12.586770 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9"] Nov 02 22:45:12.593823 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:12.593788 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a8baae-7796cfdbd4-ln6m9"] Nov 02 22:45:13.318365 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:13.318329 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01e51b93-dd50-485c-a163-6bc7387c9392" path="/var/lib/kubelet/pods/01e51b93-dd50-485c-a163-6bc7387c9392/volumes" Nov 02 22:45:13.770743 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:45:13.770704 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:45:14.377121 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:14.377088 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:14.666643 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:14.666562 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:14.952208 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:14.952095 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:45:15.472084 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:15.472046 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:16.994351 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:16.994315 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:20.406664 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:20.406626 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:21.752674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:21.752631 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:21.927435 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:21.927403 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:22.082913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:22.082805 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2"] Nov 02 22:45:22.083073 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:22.083053 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" podUID="2937a840-e872-473e-a2c7-777873cde2fb" containerName="istio-proxy" containerID="cri-o://616b7d1ffefbae322a07edfd0e992acdcb66c323a84d789321e619f3455a247e" gracePeriod=30 Nov 02 22:45:22.121438 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:22.121397 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:27.123846 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:45:27.123791 5871 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2937a840_e872_473e_a2c7_777873cde2fb.slice/crio-616b7d1ffefbae322a07edfd0e992acdcb66c323a84d789321e619f3455a247e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2937a840_e872_473e_a2c7_777873cde2fb.slice/crio-conmon-616b7d1ffefbae322a07edfd0e992acdcb66c323a84d789321e619f3455a247e.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:45:27.124262 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:45:27.123806 5871 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2937a840_e872_473e_a2c7_777873cde2fb.slice/crio-conmon-616b7d1ffefbae322a07edfd0e992acdcb66c323a84d789321e619f3455a247e.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:45:27.344672 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.344643 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:45:27.399946 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.399907 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-workload-certs\") pod \"2937a840-e872-473e-a2c7-777873cde2fb\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " Nov 02 22:45:27.400149 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400007 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2937a840-e872-473e-a2c7-777873cde2fb-kube-api-access-knb6t\" (UniqueName: \"kubernetes.io/projected/2937a840-e872-473e-a2c7-777873cde2fb-kube-api-access-knb6t\") pod \"2937a840-e872-473e-a2c7-777873cde2fb\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " Nov 02 22:45:27.400149 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400053 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2937a840-e872-473e-a2c7-777873cde2fb-istio-token\" (UniqueName: \"kubernetes.io/projected/2937a840-e872-473e-a2c7-777873cde2fb-istio-token\") pod \"2937a840-e872-473e-a2c7-777873cde2fb\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " Nov 02 22:45:27.400149 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400078 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-istio-data\") pod \"2937a840-e872-473e-a2c7-777873cde2fb\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " Nov 02 22:45:27.400149 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400099 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-workload-socket\") pod \"2937a840-e872-473e-a2c7-777873cde2fb\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " Nov 02 22:45:27.400369 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400214 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2937a840-e872-473e-a2c7-777873cde2fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2937a840-e872-473e-a2c7-777873cde2fb-istio-podinfo\") pod \"2937a840-e872-473e-a2c7-777873cde2fb\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " Nov 02 22:45:27.400369 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400254 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2937a840-e872-473e-a2c7-777873cde2fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2937a840-e872-473e-a2c7-777873cde2fb-istiod-ca-cert\") pod \"2937a840-e872-473e-a2c7-777873cde2fb\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " Nov 02 22:45:27.400369 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400265 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-workload-certs" pod "2937a840-e872-473e-a2c7-777873cde2fb" (UID: "2937a840-e872-473e-a2c7-777873cde2fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:45:27.400369 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400325 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-credential-socket\") pod \"2937a840-e872-473e-a2c7-777873cde2fb\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " Nov 02 22:45:27.400369 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400366 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-istio-envoy\") pod \"2937a840-e872-473e-a2c7-777873cde2fb\" (UID: \"2937a840-e872-473e-a2c7-777873cde2fb\") " Nov 02 22:45:27.400629 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400387 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-istio-data" pod "2937a840-e872-473e-a2c7-777873cde2fb" (UID: "2937a840-e872-473e-a2c7-777873cde2fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:45:27.400629 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400402 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-workload-socket" pod "2937a840-e872-473e-a2c7-777873cde2fb" (UID: "2937a840-e872-473e-a2c7-777873cde2fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:45:27.400629 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400605 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2937a840-e872-473e-a2c7-777873cde2fb-istiod-ca-cert" pod "2937a840-e872-473e-a2c7-777873cde2fb" (UID: "2937a840-e872-473e-a2c7-777873cde2fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:45:27.400629 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400620 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-credential-socket" pod "2937a840-e872-473e-a2c7-777873cde2fb" (UID: "2937a840-e872-473e-a2c7-777873cde2fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:45:27.400832 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400814 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:27.400913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400839 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:27.400913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400856 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2937a840-e872-473e-a2c7-777873cde2fb-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:27.400913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400894 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:27.400913 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.400909 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:27.402492 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.402469 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2937a840-e872-473e-a2c7-777873cde2fb-istio-token" pod "2937a840-e872-473e-a2c7-777873cde2fb" (UID: "2937a840-e872-473e-a2c7-777873cde2fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:45:27.402606 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.402505 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2937a840-e872-473e-a2c7-777873cde2fb-istio-podinfo" pod "2937a840-e872-473e-a2c7-777873cde2fb" (UID: "2937a840-e872-473e-a2c7-777873cde2fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:45:27.402697 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.402681 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2937a840-e872-473e-a2c7-777873cde2fb-kube-api-access-knb6t" pod "2937a840-e872-473e-a2c7-777873cde2fb" (UID: "2937a840-e872-473e-a2c7-777873cde2fb"). InnerVolumeSpecName "kube-api-access-knb6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:45:27.402765 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.402749 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-istio-envoy" pod "2937a840-e872-473e-a2c7-777873cde2fb" (UID: "2937a840-e872-473e-a2c7-777873cde2fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:45:27.473338 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.473303 5871 generic.go:356] "Generic (PLEG): container finished" podID="2937a840-e872-473e-a2c7-777873cde2fb" containerID="616b7d1ffefbae322a07edfd0e992acdcb66c323a84d789321e619f3455a247e" exitCode=0 Nov 02 22:45:27.473508 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.473353 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" Nov 02 22:45:27.473508 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.473377 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" event={"ID":"2937a840-e872-473e-a2c7-777873cde2fb","Type":"ContainerDied","Data":"616b7d1ffefbae322a07edfd0e992acdcb66c323a84d789321e619f3455a247e"} Nov 02 22:45:27.473508 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.473428 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2" event={"ID":"2937a840-e872-473e-a2c7-777873cde2fb","Type":"ContainerDied","Data":"2cdc1214f21383b1e1b98caa5c00b4dd917b105e4757e6265d54ecd0807d9d62"} Nov 02 22:45:27.473508 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.473451 5871 scope.go:122] "RemoveContainer" containerID="616b7d1ffefbae322a07edfd0e992acdcb66c323a84d789321e619f3455a247e" Nov 02 22:45:27.483262 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.483243 5871 scope.go:122] "RemoveContainer" containerID="6182aff009a0dc479ae5f3472f8f4bef2a5fc8d884b9e5af97e499332c4e74f0" Nov 02 22:45:27.493040 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.493018 5871 scope.go:122] "RemoveContainer" containerID="616b7d1ffefbae322a07edfd0e992acdcb66c323a84d789321e619f3455a247e" Nov 02 22:45:27.493370 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:45:27.493348 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616b7d1ffefbae322a07edfd0e992acdcb66c323a84d789321e619f3455a247e\": container with ID starting with 616b7d1ffefbae322a07edfd0e992acdcb66c323a84d789321e619f3455a247e not found: ID does not exist" containerID="616b7d1ffefbae322a07edfd0e992acdcb66c323a84d789321e619f3455a247e" Nov 02 22:45:27.493419 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.493381 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616b7d1ffefbae322a07edfd0e992acdcb66c323a84d789321e619f3455a247e"} err="failed to get container status \"616b7d1ffefbae322a07edfd0e992acdcb66c323a84d789321e619f3455a247e\": rpc error: code = NotFound desc = could not find container \"616b7d1ffefbae322a07edfd0e992acdcb66c323a84d789321e619f3455a247e\": container with ID starting with 616b7d1ffefbae322a07edfd0e992acdcb66c323a84d789321e619f3455a247e not found: ID does not exist" Nov 02 22:45:27.493419 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.493401 5871 scope.go:122] "RemoveContainer" containerID="6182aff009a0dc479ae5f3472f8f4bef2a5fc8d884b9e5af97e499332c4e74f0" Nov 02 22:45:27.493685 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:45:27.493664 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6182aff009a0dc479ae5f3472f8f4bef2a5fc8d884b9e5af97e499332c4e74f0\": container with ID starting with 6182aff009a0dc479ae5f3472f8f4bef2a5fc8d884b9e5af97e499332c4e74f0 not found: ID does not exist" containerID="6182aff009a0dc479ae5f3472f8f4bef2a5fc8d884b9e5af97e499332c4e74f0" Nov 02 22:45:27.493753 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.493696 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6182aff009a0dc479ae5f3472f8f4bef2a5fc8d884b9e5af97e499332c4e74f0"} err="failed to get container status \"6182aff009a0dc479ae5f3472f8f4bef2a5fc8d884b9e5af97e499332c4e74f0\": rpc error: code = NotFound desc = could not find container \"6182aff009a0dc479ae5f3472f8f4bef2a5fc8d884b9e5af97e499332c4e74f0\": container with ID starting with 6182aff009a0dc479ae5f3472f8f4bef2a5fc8d884b9e5af97e499332c4e74f0 not found: ID does not exist" Nov 02 22:45:27.502295 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.502264 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knb6t\" (UniqueName: \"kubernetes.io/projected/2937a840-e872-473e-a2c7-777873cde2fb-kube-api-access-knb6t\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:27.502295 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.502295 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2937a840-e872-473e-a2c7-777873cde2fb-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:27.502450 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.502312 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2937a840-e872-473e-a2c7-777873cde2fb-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:27.502450 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.502324 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2937a840-e872-473e-a2c7-777873cde2fb-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:45:27.629845 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.629808 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2"] Nov 02 22:45:27.634014 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:27.633983 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a8baae-openshift-default-f7456557-tdns2"] Nov 02 22:45:29.312637 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:29.312605 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2937a840-e872-473e-a2c7-777873cde2fb" path="/var/lib/kubelet/pods/2937a840-e872-473e-a2c7-777873cde2fb/volumes" Nov 02 22:45:38.369645 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:38.369604 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:45:40.011304 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:40.011256 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:40.495947 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:40.495912 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:43.770432 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:45:43.770391 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:45:46.109859 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:46.109830 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:48.626171 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:48.626130 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:45:54.906832 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:54.906789 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:45:56.479944 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:45:56.479910 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:46:13.771933 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:46:13.771868 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:46:18.303155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:46:18.303116 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:46:19.228924 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:46:19.228856 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:46:20.793063 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:46:20.793030 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:46:29.530683 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:46:29.530647 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:46:43.770318 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:46:43.770274 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:46:55.323929 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:46:55.323802 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:46:56.854383 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:46:56.854333 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:47:13.770833 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:47:13.770792 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:47:21.155614 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:47:21.155577 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:47:27.367483 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:47:27.367443 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:47:29.740081 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:47:29.740032 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tfgpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:47:29.740583 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:47:29.740133 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" podUID="76dcefe6-1e67-4ba4-98a6-9358212dee30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:47:36.380461 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:47:36.380424 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:47:37.308848 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:47:37.308813 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:47:43.769820 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:47:43.769775 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:47:44.833764 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:47:44.833719 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:47:54.578128 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:47:54.578091 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:47:55.610480 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:47:55.610443 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:47:57.179754 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:47:57.179720 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:13.606191 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:13.606152 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:13.770771 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:48:13.770734 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:48:13.958857 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:13.958816 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:48:14.110085 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.110045 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc"] Nov 02 22:48:14.110559 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.110540 5871 cpu_manager.go:414] "containerMap: removing container" podUID="2937a840-e872-473e-a2c7-777873cde2fb" containerName="istio-proxy" Nov 02 22:48:14.110664 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.110562 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2937a840-e872-473e-a2c7-777873cde2fb" containerName="istio-proxy" Nov 02 22:48:14.110664 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.110575 5871 cpu_manager.go:414] "containerMap: removing container" podUID="2937a840-e872-473e-a2c7-777873cde2fb" containerName="istio-proxy" Nov 02 22:48:14.110664 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.110581 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2937a840-e872-473e-a2c7-777873cde2fb" containerName="istio-proxy" Nov 02 22:48:14.110664 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.110597 5871 cpu_manager.go:414] "containerMap: removing container" podUID="01e51b93-dd50-485c-a163-6bc7387c9392" containerName="maas-api" Nov 02 22:48:14.110664 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.110605 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e51b93-dd50-485c-a163-6bc7387c9392" containerName="maas-api" Nov 02 22:48:14.110854 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.110694 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="2937a840-e872-473e-a2c7-777873cde2fb" containerName="istio-proxy" Nov 02 22:48:14.110854 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.110707 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e51b93-dd50-485c-a163-6bc7387c9392" containerName="maas-api" Nov 02 22:48:14.117712 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.116176 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.174240 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.174203 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-c86709-openshift-default-dockercfg-zwtjj\"" Nov 02 22:48:14.224125 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.224042 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc"] Nov 02 22:48:14.238838 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.238797 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b5a043-a382-49ee-9de6-27cf546804cb-istio-podinfo\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.239043 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.238923 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b5a043-a382-49ee-9de6-27cf546804cb-istio-token\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.239043 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.238966 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-workload-socket\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.239043 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.238985 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4dkn\" (UniqueName: \"kubernetes.io/projected/99b5a043-a382-49ee-9de6-27cf546804cb-kube-api-access-x4dkn\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.239226 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.239079 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-istio-data\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.239226 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.239198 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-workload-certs\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.239297 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.239245 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-credential-socket\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.239297 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.239274 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-istio-envoy\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.239403 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.239307 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b5a043-a382-49ee-9de6-27cf546804cb-istiod-ca-cert\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.267842 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.267799 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:14.340936 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.340868 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-workload-certs\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.341156 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.340947 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-credential-socket\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.341156 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.340974 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-istio-envoy\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.341156 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.341026 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b5a043-a382-49ee-9de6-27cf546804cb-istiod-ca-cert\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.341430 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.341406 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-workload-certs\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.341430 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.341417 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-credential-socket\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.341806 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.341784 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b5a043-a382-49ee-9de6-27cf546804cb-istiod-ca-cert\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.341926 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.341859 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b5a043-a382-49ee-9de6-27cf546804cb-istio-podinfo\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.342002 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.341928 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b5a043-a382-49ee-9de6-27cf546804cb-istio-token\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.342065 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.341982 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-workload-socket\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.342065 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.342041 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4dkn\" (UniqueName: \"kubernetes.io/projected/99b5a043-a382-49ee-9de6-27cf546804cb-kube-api-access-x4dkn\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.342335 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.342307 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-workload-socket\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.342495 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.342395 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-istio-data\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.342689 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.342665 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-istio-data\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.343711 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.343678 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-istio-envoy\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.344550 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.344526 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b5a043-a382-49ee-9de6-27cf546804cb-istio-podinfo\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.404176 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.404139 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4dkn\" (UniqueName: \"kubernetes.io/projected/99b5a043-a382-49ee-9de6-27cf546804cb-kube-api-access-x4dkn\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.404579 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.404542 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b5a043-a382-49ee-9de6-27cf546804cb-istio-token\") pod \"e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.433720 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.433677 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:14.446480 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.446443 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:14.875730 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.875702 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc"] Nov 02 22:48:14.878836 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:48:14.878808 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b5a043_a382_49ee_9de6_27cf546804cb.slice/crio-e2e9949855a0011650e9c425642a160e4805984340e151bfc58c8c178b5b8ae4 WatchSource:0}: Error finding container e2e9949855a0011650e9c425642a160e4805984340e151bfc58c8c178b5b8ae4: Status 404 returned error can't find the container with id e2e9949855a0011650e9c425642a160e4805984340e151bfc58c8c178b5b8ae4 Nov 02 22:48:14.881449 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.881415 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:48:14.881537 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.881501 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:48:14.881605 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:14.881546 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:48:15.438699 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:15.438658 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" event={"ID":"99b5a043-a382-49ee-9de6-27cf546804cb","Type":"ContainerStarted","Data":"4308ab0127c13cb021286da84ee1cd38618805451720381251f35b35d8de9f74"} Nov 02 22:48:15.438920 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:15.438722 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" event={"ID":"99b5a043-a382-49ee-9de6-27cf546804cb","Type":"ContainerStarted","Data":"e2e9949855a0011650e9c425642a160e4805984340e151bfc58c8c178b5b8ae4"} Nov 02 22:48:15.562144 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:15.562090 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" podStartSLOduration=1.562074508 podStartE2EDuration="1.562074508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:15.559942164 +0000 UTC m=+2518.372272341" watchObservedRunningTime="2026-11-02 22:48:15.562074508 +0000 UTC m=+2518.374404685" Nov 02 22:48:16.434207 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:16.434174 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:16.439192 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:16.439163 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:16.444302 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:16.444280 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:16.445128 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:16.445110 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:18.581086 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:18.581024 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:18.775358 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:18.775314 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:18.956918 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:18.956836 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:21.510428 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:21.510385 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:24.898770 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:24.898681 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9"] Nov 02 22:48:24.899203 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:24.899188 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="2937a840-e872-473e-a2c7-777873cde2fb" containerName="istio-proxy" Nov 02 22:48:24.902646 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:24.902626 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:24.955392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:24.955351 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-c86709-serving-cert\"" Nov 02 22:48:24.957897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:24.957844 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0cf409-1d4c-4d3f-8302-2d7be495466f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:24.957897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:24.957892 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwwkd\" (UniqueName: \"kubernetes.io/projected/1e0cf409-1d4c-4d3f-8302-2d7be495466f-kube-api-access-lwwkd\") pod \"maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:24.958099 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:24.958074 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:24.958134 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:24.958115 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:25.007670 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:25.007628 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9"] Nov 02 22:48:25.058575 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:25.058539 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:25.058757 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:25.058580 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:25.058757 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:25.058667 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0cf409-1d4c-4d3f-8302-2d7be495466f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:25.058757 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:25.058685 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwwkd\" (UniqueName: \"kubernetes.io/projected/1e0cf409-1d4c-4d3f-8302-2d7be495466f-kube-api-access-lwwkd\") pod \"maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:25.058947 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:48:25.058759 5871 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-serving-cert: secret "maas-api-e2e-worker-single-main-c86709-serving-cert" not found Nov 02 22:48:25.058947 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:48:25.058840 5871 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-maas-api-tls podName:1e0cf409-1d4c-4d3f-8302-2d7be495466f nodeName:}" failed. No retries permitted until 2026-11-02 22:48:25.558816856 +0000 UTC m=+2528.371147033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-maas-api-tls") pod "maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" (UID: "1e0cf409-1d4c-4d3f-8302-2d7be495466f") : secret "maas-api-e2e-worker-single-main-c86709-serving-cert" not found Nov 02 22:48:25.059339 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:25.059314 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0cf409-1d4c-4d3f-8302-2d7be495466f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:25.061196 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:25.061174 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:25.120447 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:25.120411 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwwkd\" (UniqueName: \"kubernetes.io/projected/1e0cf409-1d4c-4d3f-8302-2d7be495466f-kube-api-access-lwwkd\") pod \"maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:25.131078 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:25.131047 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:25.313441 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:25.313325 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:25.562866 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:25.562816 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:25.565349 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:25.565284 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:25.816560 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:25.816471 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:26.198726 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:26.198689 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9"] Nov 02 22:48:26.201995 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:48:26.201965 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0cf409_1d4c_4d3f_8302_2d7be495466f.slice/crio-92568d71944f576e833c438de58cc948de9f3ad049f18eb8cc22225dc465fdb3 WatchSource:0}: Error finding container 92568d71944f576e833c438de58cc948de9f3ad049f18eb8cc22225dc465fdb3: Status 404 returned error can't find the container with id 92568d71944f576e833c438de58cc948de9f3ad049f18eb8cc22225dc465fdb3 Nov 02 22:48:26.499094 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:26.499012 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" event={"ID":"1e0cf409-1d4c-4d3f-8302-2d7be495466f","Type":"ContainerStarted","Data":"92568d71944f576e833c438de58cc948de9f3ad049f18eb8cc22225dc465fdb3"} Nov 02 22:48:27.506201 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:27.506093 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" event={"ID":"1e0cf409-1d4c-4d3f-8302-2d7be495466f","Type":"ContainerStarted","Data":"92e094f03b75d963efd15155233eecac48b47e109ac34d2e42b1ddeea06dafe6"} Nov 02 22:48:27.506633 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:27.506305 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:27.625322 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:27.625257 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" podStartSLOduration=2.682316026 podStartE2EDuration="3.625242558s" totalImagesPullingTime="942.926532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:48:27.624321894 +0000 UTC m=+2530.436652073" watchObservedRunningTime="2026-11-02 22:48:27.625242558 +0000 UTC m=+2530.437572733" Nov 02 22:48:30.282004 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:30.281965 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:30.784886 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:30.784836 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:30.959471 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:30.959440 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:33.265394 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:33.265344 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:48:33.518066 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:33.517980 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:48:35.509583 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:35.509546 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:40.504859 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:40.504799 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:43.374449 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:43.374397 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:43.618036 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:43.617997 5871 generic.go:356] "Generic (PLEG): container finished" podID="99b5a043-a382-49ee-9de6-27cf546804cb" containerID="4308ab0127c13cb021286da84ee1cd38618805451720381251f35b35d8de9f74" exitCode=0 Nov 02 22:48:43.618253 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:43.618069 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" event={"ID":"99b5a043-a382-49ee-9de6-27cf546804cb","Type":"ContainerDied","Data":"4308ab0127c13cb021286da84ee1cd38618805451720381251f35b35d8de9f74"} Nov 02 22:48:43.618509 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:43.618495 5871 scope.go:122] "RemoveContainer" containerID="4308ab0127c13cb021286da84ee1cd38618805451720381251f35b35d8de9f74" Nov 02 22:48:43.619722 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:43.619692 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:48:43.619823 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:43.619761 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:48:43.619823 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:43.619793 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:48:43.770680 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:48:43.770635 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:48:44.002024 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:44.001937 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:44.185197 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:44.185154 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:44.626264 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:44.626230 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" event={"ID":"99b5a043-a382-49ee-9de6-27cf546804cb","Type":"ContainerStarted","Data":"8f592c7edb6722fc67e7d89a5f0fcfe43ac60ecf65abcd79bdb4c9c3663edecf"} Nov 02 22:48:45.434789 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:45.434752 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:45.436034 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:45.435995 5871 patch_prober.go:29] interesting pod/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:48:45.436194 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:45.436047 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" podUID="99b5a043-a382-49ee-9de6-27cf546804cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:48:45.788456 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:45.788341 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:46.434221 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:46.434179 5871 patch_prober.go:29] interesting pod/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:48:46.434405 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:46.434253 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" podUID="99b5a043-a382-49ee-9de6-27cf546804cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:48:46.444240 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:46.444209 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:46.805368 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:46.805249 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:47.434460 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:47.434421 5871 patch_prober.go:29] interesting pod/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:48:47.434658 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:47.434479 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" podUID="99b5a043-a382-49ee-9de6-27cf546804cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:48:48.389276 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:48.389191 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:48.434626 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:48.434581 5871 patch_prober.go:29] interesting pod/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:48:48.434802 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:48.434645 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" podUID="99b5a043-a382-49ee-9de6-27cf546804cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:48:49.434275 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:49.434234 5871 patch_prober.go:29] interesting pod/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:48:49.434705 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:49.434300 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" podUID="99b5a043-a382-49ee-9de6-27cf546804cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:48:50.439177 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:50.439140 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:50.440285 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:50.440260 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:48:50.685118 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:50.685063 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:51.460227 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:51.460189 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:48:55.709508 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:55.709474 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:55.944196 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:55.944138 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:48:57.534271 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:48:57.534236 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:00.676482 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:00.676448 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:05.701619 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:05.701577 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:10.720135 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:10.720096 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:13.770934 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:49:13.770863 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:49:15.691342 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:15.691305 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:19.533154 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:19.533117 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:49:20.682421 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:20.682381 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:21.845637 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:21.845601 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:25.696241 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:25.696201 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:30.689711 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:30.689675 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:35.707587 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:35.707546 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:39.003136 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:39.003104 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:39.185644 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:39.185610 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:40.596292 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:40.596259 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:43.674367 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:43.674332 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:43.770665 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:49:43.770624 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:49:45.363360 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:45.363326 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:49:45.741597 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:45.741501 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:48.713275 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:48.713213 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:50.991745 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:50.991695 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:55.202126 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:55.202028 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9"] Nov 02 22:49:55.202612 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:55.202335 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" podUID="1e0cf409-1d4c-4d3f-8302-2d7be495466f" containerName="maas-api" containerID="cri-o://92e094f03b75d963efd15155233eecac48b47e109ac34d2e42b1ddeea06dafe6" gracePeriod=30 Nov 02 22:49:55.334540 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:55.334487 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:55.860399 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:55.860369 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:56.036067 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:56.036025 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:56.280619 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:56.280596 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:56.464388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:56.464360 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:49:56.540734 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:56.540651 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-metrics-certs\") pod \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " Nov 02 22:49:56.540952 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:56.540736 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0cf409-1d4c-4d3f-8302-2d7be495466f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0cf409-1d4c-4d3f-8302-2d7be495466f-openshift-service-ca\") pod \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " Nov 02 22:49:56.540952 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:56.540803 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0cf409-1d4c-4d3f-8302-2d7be495466f-kube-api-access-lwwkd\" (UniqueName: \"kubernetes.io/projected/1e0cf409-1d4c-4d3f-8302-2d7be495466f-kube-api-access-lwwkd\") pod \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " Nov 02 22:49:56.540952 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:56.540827 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-maas-api-tls\") pod \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\" (UID: \"1e0cf409-1d4c-4d3f-8302-2d7be495466f\") " Nov 02 22:49:56.541187 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:56.541160 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0cf409-1d4c-4d3f-8302-2d7be495466f-openshift-service-ca" pod "1e0cf409-1d4c-4d3f-8302-2d7be495466f" (UID: "1e0cf409-1d4c-4d3f-8302-2d7be495466f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:49:56.541367 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:56.541347 5871 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0cf409-1d4c-4d3f-8302-2d7be495466f-openshift-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:49:56.543134 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:56.543103 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-maas-api-tls" pod "1e0cf409-1d4c-4d3f-8302-2d7be495466f" (UID: "1e0cf409-1d4c-4d3f-8302-2d7be495466f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:56.543256 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:56.543127 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-metrics-certs" pod "1e0cf409-1d4c-4d3f-8302-2d7be495466f" (UID: "1e0cf409-1d4c-4d3f-8302-2d7be495466f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:49:56.543256 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:56.543128 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0cf409-1d4c-4d3f-8302-2d7be495466f-kube-api-access-lwwkd" pod "1e0cf409-1d4c-4d3f-8302-2d7be495466f" (UID: "1e0cf409-1d4c-4d3f-8302-2d7be495466f"). InnerVolumeSpecName "kube-api-access-lwwkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:49:56.642627 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:56.642587 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwwkd\" (UniqueName: \"kubernetes.io/projected/1e0cf409-1d4c-4d3f-8302-2d7be495466f-kube-api-access-lwwkd\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:49:56.642627 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:56.642622 5871 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-maas-api-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:49:56.642627 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:56.642632 5871 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0cf409-1d4c-4d3f-8302-2d7be495466f-metrics-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:49:57.051685 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:57.051646 5871 generic.go:356] "Generic (PLEG): container finished" podID="1e0cf409-1d4c-4d3f-8302-2d7be495466f" containerID="92e094f03b75d963efd15155233eecac48b47e109ac34d2e42b1ddeea06dafe6" exitCode=0 Nov 02 22:49:57.051886 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:57.051700 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" Nov 02 22:49:57.051886 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:57.051727 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" event={"ID":"1e0cf409-1d4c-4d3f-8302-2d7be495466f","Type":"ContainerDied","Data":"92e094f03b75d963efd15155233eecac48b47e109ac34d2e42b1ddeea06dafe6"} Nov 02 22:49:57.051886 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:57.051788 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9" event={"ID":"1e0cf409-1d4c-4d3f-8302-2d7be495466f","Type":"ContainerDied","Data":"92568d71944f576e833c438de58cc948de9f3ad049f18eb8cc22225dc465fdb3"} Nov 02 22:49:57.051886 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:57.051813 5871 scope.go:122] "RemoveContainer" containerID="92e094f03b75d963efd15155233eecac48b47e109ac34d2e42b1ddeea06dafe6" Nov 02 22:49:57.062594 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:57.062565 5871 scope.go:122] "RemoveContainer" containerID="92e094f03b75d963efd15155233eecac48b47e109ac34d2e42b1ddeea06dafe6" Nov 02 22:49:57.062924 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:49:57.062899 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92e094f03b75d963efd15155233eecac48b47e109ac34d2e42b1ddeea06dafe6\": container with ID starting with 92e094f03b75d963efd15155233eecac48b47e109ac34d2e42b1ddeea06dafe6 not found: ID does not exist" containerID="92e094f03b75d963efd15155233eecac48b47e109ac34d2e42b1ddeea06dafe6" Nov 02 22:49:57.062984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:57.062937 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e094f03b75d963efd15155233eecac48b47e109ac34d2e42b1ddeea06dafe6"} err="failed to get container status \"92e094f03b75d963efd15155233eecac48b47e109ac34d2e42b1ddeea06dafe6\": rpc error: code = NotFound desc = could not find container \"92e094f03b75d963efd15155233eecac48b47e109ac34d2e42b1ddeea06dafe6\": container with ID starting with 92e094f03b75d963efd15155233eecac48b47e109ac34d2e42b1ddeea06dafe6 not found: ID does not exist" Nov 02 22:49:57.217036 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:57.217000 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9"] Nov 02 22:49:57.266800 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:57.266765 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c86709-7b86bc6785-fzgq9"] Nov 02 22:49:57.311868 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:57.311786 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0cf409-1d4c-4d3f-8302-2d7be495466f" path="/var/lib/kubelet/pods/1e0cf409-1d4c-4d3f-8302-2d7be495466f/volumes" Nov 02 22:49:59.668762 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:59.668727 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:59.950366 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:59.950278 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:49:59.959732 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:49:59.959695 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:50:00.830113 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:00.830077 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:06.650511 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:06.650474 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:06.829840 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:06.829794 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:06.969549 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:06.969464 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc"] Nov 02 22:50:06.969714 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:06.969683 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" podUID="99b5a043-a382-49ee-9de6-27cf546804cb" containerName="istio-proxy" containerID="cri-o://8f592c7edb6722fc67e7d89a5f0fcfe43ac60ecf65abcd79bdb4c9c3663edecf" gracePeriod=30 Nov 02 22:50:07.017224 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:07.017159 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:12.140026 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.139633 5871 generic.go:356] "Generic (PLEG): container finished" podID="99b5a043-a382-49ee-9de6-27cf546804cb" containerID="8f592c7edb6722fc67e7d89a5f0fcfe43ac60ecf65abcd79bdb4c9c3663edecf" exitCode=0 Nov 02 22:50:12.140026 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.139988 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" event={"ID":"99b5a043-a382-49ee-9de6-27cf546804cb","Type":"ContainerDied","Data":"8f592c7edb6722fc67e7d89a5f0fcfe43ac60ecf65abcd79bdb4c9c3663edecf"} Nov 02 22:50:12.140425 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.140055 5871 scope.go:122] "RemoveContainer" containerID="4308ab0127c13cb021286da84ee1cd38618805451720381251f35b35d8de9f74" Nov 02 22:50:12.227388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.227358 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:50:12.399237 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.399199 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99b5a043-a382-49ee-9de6-27cf546804cb-istio-token\" (UniqueName: \"kubernetes.io/projected/99b5a043-a382-49ee-9de6-27cf546804cb-istio-token\") pod \"99b5a043-a382-49ee-9de6-27cf546804cb\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " Nov 02 22:50:12.399459 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.399285 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99b5a043-a382-49ee-9de6-27cf546804cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b5a043-a382-49ee-9de6-27cf546804cb-istiod-ca-cert\") pod \"99b5a043-a382-49ee-9de6-27cf546804cb\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " Nov 02 22:50:12.399459 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.399376 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-istio-data\") pod \"99b5a043-a382-49ee-9de6-27cf546804cb\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " Nov 02 22:50:12.399587 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.399468 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-workload-socket\") pod \"99b5a043-a382-49ee-9de6-27cf546804cb\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " Nov 02 22:50:12.399587 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.399498 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-istio-envoy\") pod \"99b5a043-a382-49ee-9de6-27cf546804cb\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " Nov 02 22:50:12.399587 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.399542 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-credential-socket\") pod \"99b5a043-a382-49ee-9de6-27cf546804cb\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " Nov 02 22:50:12.399757 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.399615 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99b5a043-a382-49ee-9de6-27cf546804cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b5a043-a382-49ee-9de6-27cf546804cb-istio-podinfo\") pod \"99b5a043-a382-49ee-9de6-27cf546804cb\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " Nov 02 22:50:12.399757 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.399642 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-workload-certs\") pod \"99b5a043-a382-49ee-9de6-27cf546804cb\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " Nov 02 22:50:12.399757 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.399681 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99b5a043-a382-49ee-9de6-27cf546804cb-kube-api-access-x4dkn\" (UniqueName: \"kubernetes.io/projected/99b5a043-a382-49ee-9de6-27cf546804cb-kube-api-access-x4dkn\") pod \"99b5a043-a382-49ee-9de6-27cf546804cb\" (UID: \"99b5a043-a382-49ee-9de6-27cf546804cb\") " Nov 02 22:50:12.399757 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.399727 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-istio-data" pod "99b5a043-a382-49ee-9de6-27cf546804cb" (UID: "99b5a043-a382-49ee-9de6-27cf546804cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:50:12.400018 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.399885 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-credential-socket" pod "99b5a043-a382-49ee-9de6-27cf546804cb" (UID: "99b5a043-a382-49ee-9de6-27cf546804cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:50:12.400141 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.400106 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-workload-certs" pod "99b5a043-a382-49ee-9de6-27cf546804cb" (UID: "99b5a043-a382-49ee-9de6-27cf546804cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:50:12.400372 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.400341 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b5a043-a382-49ee-9de6-27cf546804cb-istiod-ca-cert" pod "99b5a043-a382-49ee-9de6-27cf546804cb" (UID: "99b5a043-a382-49ee-9de6-27cf546804cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:50:12.401853 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.401665 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-workload-socket" pod "99b5a043-a382-49ee-9de6-27cf546804cb" (UID: "99b5a043-a382-49ee-9de6-27cf546804cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:50:12.402466 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.402442 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99b5a043-a382-49ee-9de6-27cf546804cb-istio-podinfo" pod "99b5a043-a382-49ee-9de6-27cf546804cb" (UID: "99b5a043-a382-49ee-9de6-27cf546804cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:50:12.402466 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.402458 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b5a043-a382-49ee-9de6-27cf546804cb-istio-token" pod "99b5a043-a382-49ee-9de6-27cf546804cb" (UID: "99b5a043-a382-49ee-9de6-27cf546804cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:50:12.402683 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.402646 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b5a043-a382-49ee-9de6-27cf546804cb-kube-api-access-x4dkn" pod "99b5a043-a382-49ee-9de6-27cf546804cb" (UID: "99b5a043-a382-49ee-9de6-27cf546804cb"). InnerVolumeSpecName "kube-api-access-x4dkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:50:12.403120 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.403093 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-istio-envoy" pod "99b5a043-a382-49ee-9de6-27cf546804cb" (UID: "99b5a043-a382-49ee-9de6-27cf546804cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:50:12.403601 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.403577 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:12.403641 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.403618 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:12.504363 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.504327 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:12.504363 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.504356 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:12.504363 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.504367 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b5a043-a382-49ee-9de6-27cf546804cb-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:12.504602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.504375 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b5a043-a382-49ee-9de6-27cf546804cb-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:12.504602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.504385 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4dkn\" (UniqueName: \"kubernetes.io/projected/99b5a043-a382-49ee-9de6-27cf546804cb-kube-api-access-x4dkn\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:12.504602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.504394 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b5a043-a382-49ee-9de6-27cf546804cb-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:12.504602 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:12.504404 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b5a043-a382-49ee-9de6-27cf546804cb-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:50:13.146641 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.146610 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" Nov 02 22:50:13.147110 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.146649 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc" event={"ID":"99b5a043-a382-49ee-9de6-27cf546804cb","Type":"ContainerDied","Data":"e2e9949855a0011650e9c425642a160e4805984340e151bfc58c8c178b5b8ae4"} Nov 02 22:50:13.147110 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.146722 5871 scope.go:122] "RemoveContainer" containerID="8f592c7edb6722fc67e7d89a5f0fcfe43ac60ecf65abcd79bdb4c9c3663edecf" Nov 02 22:50:13.363883 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.363844 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc"] Nov 02 22:50:13.414504 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.414405 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c86709-openshift-default-744587df7-8gqpc"] Nov 02 22:50:13.416594 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.416565 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc"] Nov 02 22:50:13.416989 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.416976 5871 cpu_manager.go:414] "containerMap: removing container" podUID="99b5a043-a382-49ee-9de6-27cf546804cb" containerName="istio-proxy" Nov 02 22:50:13.417048 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.416991 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99b5a043-a382-49ee-9de6-27cf546804cb" containerName="istio-proxy" Nov 02 22:50:13.417048 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.417010 5871 cpu_manager.go:414] "containerMap: removing container" podUID="99b5a043-a382-49ee-9de6-27cf546804cb" containerName="istio-proxy" Nov 02 22:50:13.417048 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.417015 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99b5a043-a382-49ee-9de6-27cf546804cb" containerName="istio-proxy" Nov 02 22:50:13.417048 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.417033 5871 cpu_manager.go:414] "containerMap: removing container" podUID="1e0cf409-1d4c-4d3f-8302-2d7be495466f" containerName="maas-api" Nov 02 22:50:13.417048 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.417038 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0cf409-1d4c-4d3f-8302-2d7be495466f" containerName="maas-api" Nov 02 22:50:13.417200 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.417097 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="99b5a043-a382-49ee-9de6-27cf546804cb" containerName="istio-proxy" Nov 02 22:50:13.417200 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.417107 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="99b5a043-a382-49ee-9de6-27cf546804cb" containerName="istio-proxy" Nov 02 22:50:13.417200 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.417117 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0cf409-1d4c-4d3f-8302-2d7be495466f" containerName="maas-api" Nov 02 22:50:13.421779 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.421757 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.470057 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.470022 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc"] Nov 02 22:50:13.475341 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.475317 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e2817aa1-kserve-self-signed-certs\"" Nov 02 22:50:13.513289 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.513250 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:13.516518 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.516490 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-model-cache\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.516648 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.516526 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92e1be33-fa55-4879-802e-4f68d3315fb2-tls-certs\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.516648 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.516556 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk77j\" (UniqueName: \"kubernetes.io/projected/92e1be33-fa55-4879-802e-4f68d3315fb2-kube-api-access-pk77j\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.516724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.516643 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-dshm\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.516724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.516699 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-tmp-dir\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.516790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.516776 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-home\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.617536 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.617495 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-tmp-dir\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.617691 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.617642 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-home\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.617755 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.617715 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-model-cache\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.617755 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.617742 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92e1be33-fa55-4879-802e-4f68d3315fb2-tls-certs\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.617850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.617773 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk77j\" (UniqueName: \"kubernetes.io/projected/92e1be33-fa55-4879-802e-4f68d3315fb2-kube-api-access-pk77j\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.617953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.617924 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-dshm\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.618029 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.617935 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-tmp-dir\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.618092 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.618050 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-home\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.618208 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.618189 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-model-cache\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.620139 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.620112 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-dshm\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.620292 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.620272 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92e1be33-fa55-4879-802e-4f68d3315fb2-tls-certs\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.679377 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.679300 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk77j\" (UniqueName: \"kubernetes.io/projected/92e1be33-fa55-4879-802e-4f68d3315fb2-kube-api-access-pk77j\") pod \"e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.698806 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.698768 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:13.738487 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.738446 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:13.773207 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:50:13.773160 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:50:13.873902 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:13.873844 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:14.125658 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:14.125628 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc"] Nov 02 22:50:14.128796 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:50:14.128767 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92e1be33_fa55_4879_802e_4f68d3315fb2.slice/crio-0583964c18ae8f6c132a0d3e7ff227c477fefe23995ab7fa58e4ea36b417f6c5 WatchSource:0}: Error finding container 0583964c18ae8f6c132a0d3e7ff227c477fefe23995ab7fa58e4ea36b417f6c5: Status 404 returned error can't find the container with id 0583964c18ae8f6c132a0d3e7ff227c477fefe23995ab7fa58e4ea36b417f6c5 Nov 02 22:50:14.153846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:14.153818 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" event={"ID":"92e1be33-fa55-4879-802e-4f68d3315fb2","Type":"ContainerStarted","Data":"0583964c18ae8f6c132a0d3e7ff227c477fefe23995ab7fa58e4ea36b417f6c5"} Nov 02 22:50:15.160211 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:15.160175 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" event={"ID":"92e1be33-fa55-4879-802e-4f68d3315fb2","Type":"ContainerStarted","Data":"076d3dfcb62e864784c53e21e0ad84a579b9ad875355869378dde1721d9adda3"} Nov 02 22:50:15.279819 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:15.279759 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" podStartSLOduration=2.279744307 podStartE2EDuration="2.279744307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:50:15.277625065 +0000 UTC m=+2638.089955245" watchObservedRunningTime="2026-11-02 22:50:15.279744307 +0000 UTC m=+2638.092074485" Nov 02 22:50:15.312489 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:15.312453 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99b5a043-a382-49ee-9de6-27cf546804cb" path="/var/lib/kubelet/pods/99b5a043-a382-49ee-9de6-27cf546804cb/volumes" Nov 02 22:50:15.739045 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:15.739011 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:17.975508 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:17.975471 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:50:22.192397 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:22.192302 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:23.739768 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:23.739729 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:23.752466 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:23.752433 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:24.251255 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:24.251219 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:50:26.014257 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:26.014220 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:26.321273 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:26.321179 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:40.974281 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:40.974246 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:43.770376 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:50:43.770335 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:50:53.907256 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:53.907225 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:50:58.545225 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:50:58.545190 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:51:02.950822 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:02.950762 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:51:13.770547 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:51:13.770506 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:51:22.537904 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:22.537840 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:51:26.673263 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:26.673167 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:51:29.747086 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:29.747037 5871 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tfgpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:51:29.747516 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:29.747123 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tfgpj" podUID="76dcefe6-1e67-4ba4-98a6-9358212dee30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:51:36.792763 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:36.792722 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:51:36.971206 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:36.971165 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:51:38.491054 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:38.491021 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc"] Nov 02 22:51:38.538634 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:38.538598 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:51:42.659739 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:42.659703 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:51:43.773077 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:51:43.773034 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:51:46.863070 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:46.863026 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:51:53.516897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:53.515069 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" podUID="92e1be33-fa55-4879-802e-4f68d3315fb2" containerName="main" containerID="cri-o://076d3dfcb62e864784c53e21e0ad84a579b9ad875355869378dde1721d9adda3" gracePeriod=45 Nov 02 22:51:53.842748 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:53.842656 5871 generic.go:356] "Generic (PLEG): container finished" podID="92e1be33-fa55-4879-802e-4f68d3315fb2" containerID="076d3dfcb62e864784c53e21e0ad84a579b9ad875355869378dde1721d9adda3" exitCode=0 Nov 02 22:51:53.842748 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:53.842725 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" event={"ID":"92e1be33-fa55-4879-802e-4f68d3315fb2","Type":"ContainerDied","Data":"076d3dfcb62e864784c53e21e0ad84a579b9ad875355869378dde1721d9adda3"} Nov 02 22:51:54.026653 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.026625 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:51:54.135897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.135470 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-dshm\") pod \"92e1be33-fa55-4879-802e-4f68d3315fb2\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " Nov 02 22:51:54.135897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.135554 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-tmp-dir\") pod \"92e1be33-fa55-4879-802e-4f68d3315fb2\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " Nov 02 22:51:54.135897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.135597 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-home\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-home\") pod \"92e1be33-fa55-4879-802e-4f68d3315fb2\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " Nov 02 22:51:54.135897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.135647 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-model-cache\") pod \"92e1be33-fa55-4879-802e-4f68d3315fb2\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " Nov 02 22:51:54.135897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.135797 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e1be33-fa55-4879-802e-4f68d3315fb2-kube-api-access-pk77j\" (UniqueName: \"kubernetes.io/projected/92e1be33-fa55-4879-802e-4f68d3315fb2-kube-api-access-pk77j\") pod \"92e1be33-fa55-4879-802e-4f68d3315fb2\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " Nov 02 22:51:54.135897 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.135830 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92e1be33-fa55-4879-802e-4f68d3315fb2-tls-certs\" (UniqueName: \"kubernetes.io/secret/92e1be33-fa55-4879-802e-4f68d3315fb2-tls-certs\") pod \"92e1be33-fa55-4879-802e-4f68d3315fb2\" (UID: \"92e1be33-fa55-4879-802e-4f68d3315fb2\") " Nov 02 22:51:54.136639 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.136611 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-home" pod "92e1be33-fa55-4879-802e-4f68d3315fb2" (UID: "92e1be33-fa55-4879-802e-4f68d3315fb2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:51:54.137020 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.136858 5871 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-home\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:51:54.137194 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.137171 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-model-cache" pod "92e1be33-fa55-4879-802e-4f68d3315fb2" (UID: "92e1be33-fa55-4879-802e-4f68d3315fb2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:51:54.137486 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.137467 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-tmp-dir" pod "92e1be33-fa55-4879-802e-4f68d3315fb2" (UID: "92e1be33-fa55-4879-802e-4f68d3315fb2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:51:54.138856 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.138833 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-dshm" pod "92e1be33-fa55-4879-802e-4f68d3315fb2" (UID: "92e1be33-fa55-4879-802e-4f68d3315fb2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:51:54.139634 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.139608 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e1be33-fa55-4879-802e-4f68d3315fb2-kube-api-access-pk77j" pod "92e1be33-fa55-4879-802e-4f68d3315fb2" (UID: "92e1be33-fa55-4879-802e-4f68d3315fb2"). InnerVolumeSpecName "kube-api-access-pk77j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:51:54.139720 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.139692 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92e1be33-fa55-4879-802e-4f68d3315fb2-tls-certs" pod "92e1be33-fa55-4879-802e-4f68d3315fb2" (UID: "92e1be33-fa55-4879-802e-4f68d3315fb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:51:54.240922 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.240835 5871 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-dshm\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:51:54.241301 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.241284 5871 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-tmp-dir\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:51:54.241467 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.241455 5871 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e1be33-fa55-4879-802e-4f68d3315fb2-model-cache\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:51:54.241594 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.241580 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk77j\" (UniqueName: \"kubernetes.io/projected/92e1be33-fa55-4879-802e-4f68d3315fb2-kube-api-access-pk77j\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:51:54.241696 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.241684 5871 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92e1be33-fa55-4879-802e-4f68d3315fb2-tls-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:51:54.850407 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.850374 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" Nov 02 22:51:54.850914 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.850367 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc" event={"ID":"92e1be33-fa55-4879-802e-4f68d3315fb2","Type":"ContainerDied","Data":"0583964c18ae8f6c132a0d3e7ff227c477fefe23995ab7fa58e4ea36b417f6c5"} Nov 02 22:51:54.850914 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:54.850503 5871 scope.go:122] "RemoveContainer" containerID="076d3dfcb62e864784c53e21e0ad84a579b9ad875355869378dde1721d9adda3" Nov 02 22:51:55.016105 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:55.016067 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc"] Nov 02 22:51:55.066128 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:55.066091 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e2817aa1-kserve-55d7bb5d57-zsktc"] Nov 02 22:51:55.312908 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:55.312859 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e1be33-fa55-4879-802e-4f68d3315fb2" path="/var/lib/kubelet/pods/92e1be33-fa55-4879-802e-4f68d3315fb2/volumes" Nov 02 22:51:58.579502 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:51:58.579463 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:52:01.710043 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:01.710003 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:02.720302 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:02.720269 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:02.890814 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:02.890779 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:03.301829 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:03.301793 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:06.597985 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:06.597946 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:10.642265 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:10.642227 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:52:13.769742 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:52:13.769698 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:52:22.903952 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:22.903916 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:27.233558 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:27.233522 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:33.385969 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:33.385930 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:52:33.950461 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:33.950423 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86c65979d7-v94d5"] Nov 02 22:52:33.951120 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:33.951100 5871 cpu_manager.go:414] "containerMap: removing container" podUID="92e1be33-fa55-4879-802e-4f68d3315fb2" containerName="main" Nov 02 22:52:33.951218 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:33.951122 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e1be33-fa55-4879-802e-4f68d3315fb2" containerName="main" Nov 02 22:52:33.951257 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:33.951220 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e1be33-fa55-4879-802e-4f68d3315fb2" containerName="main" Nov 02 22:52:33.956062 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:33.956038 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" Nov 02 22:52:34.024343 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:34.024301 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a66b293a-0f29-4692-97d7-c64412b84758-openshift-service-ca\") pod \"maas-api-86c65979d7-v94d5\" (UID: \"a66b293a-0f29-4692-97d7-c64412b84758\") " pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" Nov 02 22:52:34.024534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:34.024465 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a66b293a-0f29-4692-97d7-c64412b84758-maas-api-tls\") pod \"maas-api-86c65979d7-v94d5\" (UID: \"a66b293a-0f29-4692-97d7-c64412b84758\") " pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" Nov 02 22:52:34.024534 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:34.024523 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a66b293a-0f29-4692-97d7-c64412b84758-metrics-certs\") pod \"maas-api-86c65979d7-v94d5\" (UID: \"a66b293a-0f29-4692-97d7-c64412b84758\") " pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" Nov 02 22:52:34.024689 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:34.024658 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq5vb\" (UniqueName: \"kubernetes.io/projected/a66b293a-0f29-4692-97d7-c64412b84758-kube-api-access-tq5vb\") pod \"maas-api-86c65979d7-v94d5\" (UID: \"a66b293a-0f29-4692-97d7-c64412b84758\") " pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" Nov 02 22:52:34.037247 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:34.037207 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86c65979d7-v94d5"] Nov 02 22:52:34.125710 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:34.125672 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq5vb\" (UniqueName: \"kubernetes.io/projected/a66b293a-0f29-4692-97d7-c64412b84758-kube-api-access-tq5vb\") pod \"maas-api-86c65979d7-v94d5\" (UID: \"a66b293a-0f29-4692-97d7-c64412b84758\") " pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" Nov 02 22:52:34.125908 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:34.125731 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a66b293a-0f29-4692-97d7-c64412b84758-openshift-service-ca\") pod \"maas-api-86c65979d7-v94d5\" (UID: \"a66b293a-0f29-4692-97d7-c64412b84758\") " pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" Nov 02 22:52:34.125908 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:34.125798 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a66b293a-0f29-4692-97d7-c64412b84758-maas-api-tls\") pod \"maas-api-86c65979d7-v94d5\" (UID: \"a66b293a-0f29-4692-97d7-c64412b84758\") " pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" Nov 02 22:52:34.125908 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:34.125822 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a66b293a-0f29-4692-97d7-c64412b84758-metrics-certs\") pod \"maas-api-86c65979d7-v94d5\" (UID: \"a66b293a-0f29-4692-97d7-c64412b84758\") " pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" Nov 02 22:52:34.126512 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:34.126487 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a66b293a-0f29-4692-97d7-c64412b84758-openshift-service-ca\") pod \"maas-api-86c65979d7-v94d5\" (UID: \"a66b293a-0f29-4692-97d7-c64412b84758\") " pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" Nov 02 22:52:34.128297 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:34.128273 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a66b293a-0f29-4692-97d7-c64412b84758-maas-api-tls\") pod \"maas-api-86c65979d7-v94d5\" (UID: \"a66b293a-0f29-4692-97d7-c64412b84758\") " pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" Nov 02 22:52:34.128442 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:34.128420 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a66b293a-0f29-4692-97d7-c64412b84758-metrics-certs\") pod \"maas-api-86c65979d7-v94d5\" (UID: \"a66b293a-0f29-4692-97d7-c64412b84758\") " pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" Nov 02 22:52:34.187553 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:34.187511 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq5vb\" (UniqueName: \"kubernetes.io/projected/a66b293a-0f29-4692-97d7-c64412b84758-kube-api-access-tq5vb\") pod \"maas-api-86c65979d7-v94d5\" (UID: \"a66b293a-0f29-4692-97d7-c64412b84758\") " pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" Nov 02 22:52:34.270554 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:34.270467 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" Nov 02 22:52:34.650378 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:34.650279 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86c65979d7-v94d5"] Nov 02 22:52:34.653425 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:52:34.653395 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66b293a_0f29_4692_97d7_c64412b84758.slice/crio-7ba48a92e19455254ab11d54956531d0109c25072f0a4f325d80d7339e2fb746 WatchSource:0}: Error finding container 7ba48a92e19455254ab11d54956531d0109c25072f0a4f325d80d7339e2fb746: Status 404 returned error can't find the container with id 7ba48a92e19455254ab11d54956531d0109c25072f0a4f325d80d7339e2fb746 Nov 02 22:52:34.654735 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:34.654709 5871 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:52:35.080339 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:35.080248 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" event={"ID":"a66b293a-0f29-4692-97d7-c64412b84758","Type":"ContainerStarted","Data":"7ba48a92e19455254ab11d54956531d0109c25072f0a4f325d80d7339e2fb746"} Nov 02 22:52:36.087979 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:36.087855 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" event={"ID":"a66b293a-0f29-4692-97d7-c64412b84758","Type":"ContainerStarted","Data":"42002d29c3e445f5e321379e2e196f9747d7ba3c5d0df6aef204ae1b68693eac"} Nov 02 22:52:36.088375 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:36.088032 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" Nov 02 22:52:36.207650 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:36.207597 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" podStartSLOduration=2.224161217 podStartE2EDuration="3.207582937s" totalImagesPullingTime="983.42172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:36.207176459 +0000 UTC m=+2779.019506654" watchObservedRunningTime="2026-11-02 22:52:36.207582937 +0000 UTC m=+2779.019913114" Nov 02 22:52:40.736551 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:40.736516 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:41.396865 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:41.396823 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:41.646122 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:41.646082 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:42.100836 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:42.100804 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86c65979d7-v94d5" Nov 02 22:52:42.271641 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:42.271603 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bb6546495-c4r29"] Nov 02 22:52:42.271846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:42.271810 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" podUID="6bcc67dd-503f-415d-a5b5-6ee333124562" containerName="maas-api" containerID="cri-o://158feb877db394c5c4696642f24e5a0aeaab184de3b7ff54786fbf3f53bd2167" gracePeriod=30 Nov 02 22:52:43.526042 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:43.526012 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:52:43.623784 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:43.623749 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bcc67dd-503f-415d-a5b5-6ee333124562-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bcc67dd-503f-415d-a5b5-6ee333124562-openshift-service-ca\") pod \"6bcc67dd-503f-415d-a5b5-6ee333124562\" (UID: \"6bcc67dd-503f-415d-a5b5-6ee333124562\") " Nov 02 22:52:43.623982 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:43.623814 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bcc67dd-503f-415d-a5b5-6ee333124562-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bcc67dd-503f-415d-a5b5-6ee333124562-metrics-certs\") pod \"6bcc67dd-503f-415d-a5b5-6ee333124562\" (UID: \"6bcc67dd-503f-415d-a5b5-6ee333124562\") " Nov 02 22:52:43.623982 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:43.623893 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bcc67dd-503f-415d-a5b5-6ee333124562-kube-api-access-rdbnq\" (UniqueName: \"kubernetes.io/projected/6bcc67dd-503f-415d-a5b5-6ee333124562-kube-api-access-rdbnq\") pod \"6bcc67dd-503f-415d-a5b5-6ee333124562\" (UID: \"6bcc67dd-503f-415d-a5b5-6ee333124562\") " Nov 02 22:52:43.623982 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:43.623918 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bcc67dd-503f-415d-a5b5-6ee333124562-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bcc67dd-503f-415d-a5b5-6ee333124562-maas-api-tls\") pod \"6bcc67dd-503f-415d-a5b5-6ee333124562\" (UID: \"6bcc67dd-503f-415d-a5b5-6ee333124562\") " Nov 02 22:52:43.624222 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:43.624199 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bcc67dd-503f-415d-a5b5-6ee333124562-openshift-service-ca" pod "6bcc67dd-503f-415d-a5b5-6ee333124562" (UID: "6bcc67dd-503f-415d-a5b5-6ee333124562"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:52:43.624328 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:43.624307 5871 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bcc67dd-503f-415d-a5b5-6ee333124562-openshift-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:52:43.626160 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:43.626131 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bcc67dd-503f-415d-a5b5-6ee333124562-metrics-certs" pod "6bcc67dd-503f-415d-a5b5-6ee333124562" (UID: "6bcc67dd-503f-415d-a5b5-6ee333124562"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:52:43.626160 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:43.626147 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bcc67dd-503f-415d-a5b5-6ee333124562-maas-api-tls" pod "6bcc67dd-503f-415d-a5b5-6ee333124562" (UID: "6bcc67dd-503f-415d-a5b5-6ee333124562"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:52:43.626307 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:43.626131 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bcc67dd-503f-415d-a5b5-6ee333124562-kube-api-access-rdbnq" pod "6bcc67dd-503f-415d-a5b5-6ee333124562" (UID: "6bcc67dd-503f-415d-a5b5-6ee333124562"). InnerVolumeSpecName "kube-api-access-rdbnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:52:43.725766 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:43.725714 5871 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bcc67dd-503f-415d-a5b5-6ee333124562-metrics-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:52:43.725766 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:43.725760 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdbnq\" (UniqueName: \"kubernetes.io/projected/6bcc67dd-503f-415d-a5b5-6ee333124562-kube-api-access-rdbnq\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:52:43.725766 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:43.725771 5871 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bcc67dd-503f-415d-a5b5-6ee333124562-maas-api-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:52:43.771645 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:52:43.771606 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:52:44.139592 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:44.139555 5871 generic.go:356] "Generic (PLEG): container finished" podID="6bcc67dd-503f-415d-a5b5-6ee333124562" containerID="158feb877db394c5c4696642f24e5a0aeaab184de3b7ff54786fbf3f53bd2167" exitCode=0 Nov 02 22:52:44.139779 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:44.139619 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" Nov 02 22:52:44.139779 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:44.139618 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" event={"ID":"6bcc67dd-503f-415d-a5b5-6ee333124562","Type":"ContainerDied","Data":"158feb877db394c5c4696642f24e5a0aeaab184de3b7ff54786fbf3f53bd2167"} Nov 02 22:52:44.139779 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:44.139711 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" event={"ID":"6bcc67dd-503f-415d-a5b5-6ee333124562","Type":"ContainerDied","Data":"bc90ab77d4152dca88930d767793194729be79b4af7641ea9d09820b32344509"} Nov 02 22:52:44.139779 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:44.139741 5871 scope.go:122] "RemoveContainer" containerID="158feb877db394c5c4696642f24e5a0aeaab184de3b7ff54786fbf3f53bd2167" Nov 02 22:52:44.150249 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:44.150232 5871 scope.go:122] "RemoveContainer" containerID="158feb877db394c5c4696642f24e5a0aeaab184de3b7ff54786fbf3f53bd2167" Nov 02 22:52:44.150532 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:52:44.150515 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158feb877db394c5c4696642f24e5a0aeaab184de3b7ff54786fbf3f53bd2167\": container with ID starting with 158feb877db394c5c4696642f24e5a0aeaab184de3b7ff54786fbf3f53bd2167 not found: ID does not exist" containerID="158feb877db394c5c4696642f24e5a0aeaab184de3b7ff54786fbf3f53bd2167" Nov 02 22:52:44.150579 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:44.150541 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158feb877db394c5c4696642f24e5a0aeaab184de3b7ff54786fbf3f53bd2167"} err="failed to get container status \"158feb877db394c5c4696642f24e5a0aeaab184de3b7ff54786fbf3f53bd2167\": rpc error: code = NotFound desc = could not find container \"158feb877db394c5c4696642f24e5a0aeaab184de3b7ff54786fbf3f53bd2167\": container with ID starting with 158feb877db394c5c4696642f24e5a0aeaab184de3b7ff54786fbf3f53bd2167 not found: ID does not exist" Nov 02 22:52:44.307951 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:44.307919 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bb6546495-c4r29"] Nov 02 22:52:44.312068 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:44.312045 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bb6546495-c4r29"] Nov 02 22:52:44.619888 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:44.619841 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:44.816316 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:44.816283 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:45.002133 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:45.002035 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:45.312418 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:45.312327 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bcc67dd-503f-415d-a5b5-6ee333124562" path="/var/lib/kubelet/pods/6bcc67dd-503f-415d-a5b5-6ee333124562/volumes" Nov 02 22:52:46.500764 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:46.500714 5871 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-5bb6546495-c4r29" podUID="6bcc67dd-503f-415d-a5b5-6ee333124562" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.43:8443/health\": context deadline exceeded" Nov 02 22:52:49.604022 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.603991 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99"] Nov 02 22:52:49.604434 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.604417 5871 cpu_manager.go:414] "containerMap: removing container" podUID="6bcc67dd-503f-415d-a5b5-6ee333124562" containerName="maas-api" Nov 02 22:52:49.604499 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.604436 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bcc67dd-503f-415d-a5b5-6ee333124562" containerName="maas-api" Nov 02 22:52:49.604546 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.604536 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bcc67dd-503f-415d-a5b5-6ee333124562" containerName="maas-api" Nov 02 22:52:49.609291 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.609271 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:52:49.662392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.662331 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e0efc115-serving-cert\"" Nov 02 22:52:49.688228 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.688180 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbmdg\" (UniqueName: \"kubernetes.io/projected/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-kube-api-access-dbmdg\") pod \"maas-api-e2e-ait-e0efc115-559dc69989-jkw99\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:52:49.688228 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.688233 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-metrics-certs\") pod \"maas-api-e2e-ait-e0efc115-559dc69989-jkw99\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:52:49.688447 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.688345 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-maas-api-tls\") pod \"maas-api-e2e-ait-e0efc115-559dc69989-jkw99\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:52:49.688485 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.688441 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-openshift-service-ca\") pod \"maas-api-e2e-ait-e0efc115-559dc69989-jkw99\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:52:49.715153 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.715113 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99"] Nov 02 22:52:49.789312 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.789272 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbmdg\" (UniqueName: \"kubernetes.io/projected/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-kube-api-access-dbmdg\") pod \"maas-api-e2e-ait-e0efc115-559dc69989-jkw99\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:52:49.789506 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.789319 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-metrics-certs\") pod \"maas-api-e2e-ait-e0efc115-559dc69989-jkw99\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:52:49.789506 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.789342 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-maas-api-tls\") pod \"maas-api-e2e-ait-e0efc115-559dc69989-jkw99\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:52:49.789506 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.789414 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-openshift-service-ca\") pod \"maas-api-e2e-ait-e0efc115-559dc69989-jkw99\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:52:49.789611 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:52:49.789502 5871 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-serving-cert: secret "maas-api-e2e-ait-e0efc115-serving-cert" not found Nov 02 22:52:49.789611 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:52:49.789566 5871 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-maas-api-tls podName:41d865be-b81d-4cc4-82d3-3d0d4cb0b40f nodeName:}" failed. No retries permitted until 2026-11-02 22:52:50.28954912 +0000 UTC m=+2793.101879288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-maas-api-tls") pod "maas-api-e2e-ait-e0efc115-559dc69989-jkw99" (UID: "41d865be-b81d-4cc4-82d3-3d0d4cb0b40f") : secret "maas-api-e2e-ait-e0efc115-serving-cert" not found Nov 02 22:52:49.790085 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.790067 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-openshift-service-ca\") pod \"maas-api-e2e-ait-e0efc115-559dc69989-jkw99\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:52:49.791957 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.791937 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-metrics-certs\") pod \"maas-api-e2e-ait-e0efc115-559dc69989-jkw99\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:52:49.838706 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.838674 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:49.850853 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:49.850821 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbmdg\" (UniqueName: \"kubernetes.io/projected/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-kube-api-access-dbmdg\") pod \"maas-api-e2e-ait-e0efc115-559dc69989-jkw99\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:52:50.018258 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:50.018226 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:50.193597 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:50.193556 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:50.293274 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:50.293168 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-maas-api-tls\") pod \"maas-api-e2e-ait-e0efc115-559dc69989-jkw99\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:52:50.295592 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:50.295566 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-maas-api-tls\") pod \"maas-api-e2e-ait-e0efc115-559dc69989-jkw99\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:52:50.522491 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:50.522463 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:52:50.900165 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:50.900129 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99"] Nov 02 22:52:50.903227 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:52:50.903196 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d865be_b81d_4cc4_82d3_3d0d4cb0b40f.slice/crio-3f8bb67664e88b88c86779daf3620e55aa9494c7e768d07b0673a6b2b30d149b WatchSource:0}: Error finding container 3f8bb67664e88b88c86779daf3620e55aa9494c7e768d07b0673a6b2b30d149b: Status 404 returned error can't find the container with id 3f8bb67664e88b88c86779daf3620e55aa9494c7e768d07b0673a6b2b30d149b Nov 02 22:52:51.181205 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:51.181126 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" event={"ID":"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f","Type":"ContainerStarted","Data":"3f8bb67664e88b88c86779daf3620e55aa9494c7e768d07b0673a6b2b30d149b"} Nov 02 22:52:52.188344 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:52.188304 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" event={"ID":"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f","Type":"ContainerStarted","Data":"89b2e0a253c56afe460daaf7c5d999c0f90ed577460e9dcbd1af94a1f5531924"} Nov 02 22:52:52.188761 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:52.188487 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:52:52.308151 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:52.308096 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" podStartSLOduration=2.4748510550000002 podStartE2EDuration="3.308079199s" totalImagesPullingTime="833.228144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:52.306021837 +0000 UTC m=+2795.118352018" watchObservedRunningTime="2026-11-02 22:52:52.308079199 +0000 UTC m=+2795.120409373" Nov 02 22:52:55.191132 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:55.191044 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:55.715992 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:55.715951 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:55.897383 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:55.897343 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:52:57.981589 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:57.981548 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:52:58.200094 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:52:58.200065 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:53:00.305744 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:00.305707 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:03.642453 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:03.642417 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:05.299006 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:05.298970 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:10.810983 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:10.810947 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:12.008163 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:12.008126 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:13.770215 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:53:13.770177 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:53:16.240541 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:16.240506 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:21.293610 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:21.293565 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:26.292431 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:26.292388 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:28.973060 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:28.973015 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:29.911464 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:29.911429 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:31.398173 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:31.398127 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:32.989833 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:32.989796 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:33.190588 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:33.190554 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:33.374582 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:33.374551 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:36.206800 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:36.206764 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:40.429967 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:40.429929 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99"] Nov 02 22:53:40.430457 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:40.430148 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" podUID="41d865be-b81d-4cc4-82d3-3d0d4cb0b40f" containerName="maas-api" containerID="cri-o://89b2e0a253c56afe460daaf7c5d999c0f90ed577460e9dcbd1af94a1f5531924" gracePeriod=30 Nov 02 22:53:40.564827 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:40.564791 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:40.692834 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:40.692769 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:53:40.796563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:40.796506 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-openshift-service-ca\") pod \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " Nov 02 22:53:40.796773 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:40.796581 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-maas-api-tls\") pod \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " Nov 02 22:53:40.796773 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:40.796624 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-kube-api-access-dbmdg\" (UniqueName: \"kubernetes.io/projected/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-kube-api-access-dbmdg\") pod \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " Nov 02 22:53:40.796773 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:40.796736 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-metrics-certs\") pod \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\" (UID: \"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f\") " Nov 02 22:53:40.797002 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:40.796973 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-openshift-service-ca" pod "41d865be-b81d-4cc4-82d3-3d0d4cb0b40f" (UID: "41d865be-b81d-4cc4-82d3-3d0d4cb0b40f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:53:40.797154 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:40.797136 5871 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-openshift-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:40.798892 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:40.798843 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-metrics-certs" pod "41d865be-b81d-4cc4-82d3-3d0d4cb0b40f" (UID: "41d865be-b81d-4cc4-82d3-3d0d4cb0b40f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:53:40.798892 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:40.798859 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-kube-api-access-dbmdg" pod "41d865be-b81d-4cc4-82d3-3d0d4cb0b40f" (UID: "41d865be-b81d-4cc4-82d3-3d0d4cb0b40f"). InnerVolumeSpecName "kube-api-access-dbmdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:53:40.798892 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:40.798889 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-maas-api-tls" pod "41d865be-b81d-4cc4-82d3-3d0d4cb0b40f" (UID: "41d865be-b81d-4cc4-82d3-3d0d4cb0b40f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:53:40.898455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:40.898417 5871 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-maas-api-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:40.898455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:40.898448 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbmdg\" (UniqueName: \"kubernetes.io/projected/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-kube-api-access-dbmdg\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:40.898455 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:40.898458 5871 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f-metrics-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:53:41.475731 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:41.475687 5871 generic.go:356] "Generic (PLEG): container finished" podID="41d865be-b81d-4cc4-82d3-3d0d4cb0b40f" containerID="89b2e0a253c56afe460daaf7c5d999c0f90ed577460e9dcbd1af94a1f5531924" exitCode=0 Nov 02 22:53:41.476156 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:41.475744 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" Nov 02 22:53:41.476156 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:41.475805 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" event={"ID":"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f","Type":"ContainerDied","Data":"89b2e0a253c56afe460daaf7c5d999c0f90ed577460e9dcbd1af94a1f5531924"} Nov 02 22:53:41.476156 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:41.475909 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99" event={"ID":"41d865be-b81d-4cc4-82d3-3d0d4cb0b40f","Type":"ContainerDied","Data":"3f8bb67664e88b88c86779daf3620e55aa9494c7e768d07b0673a6b2b30d149b"} Nov 02 22:53:41.476156 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:41.475938 5871 scope.go:122] "RemoveContainer" containerID="89b2e0a253c56afe460daaf7c5d999c0f90ed577460e9dcbd1af94a1f5531924" Nov 02 22:53:41.485331 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:41.485303 5871 scope.go:122] "RemoveContainer" containerID="89b2e0a253c56afe460daaf7c5d999c0f90ed577460e9dcbd1af94a1f5531924" Nov 02 22:53:41.485673 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:53:41.485646 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b2e0a253c56afe460daaf7c5d999c0f90ed577460e9dcbd1af94a1f5531924\": container with ID starting with 89b2e0a253c56afe460daaf7c5d999c0f90ed577460e9dcbd1af94a1f5531924 not found: ID does not exist" containerID="89b2e0a253c56afe460daaf7c5d999c0f90ed577460e9dcbd1af94a1f5531924" Nov 02 22:53:41.485743 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:41.485683 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b2e0a253c56afe460daaf7c5d999c0f90ed577460e9dcbd1af94a1f5531924"} err="failed to get container status \"89b2e0a253c56afe460daaf7c5d999c0f90ed577460e9dcbd1af94a1f5531924\": rpc error: code = NotFound desc = could not find container \"89b2e0a253c56afe460daaf7c5d999c0f90ed577460e9dcbd1af94a1f5531924\": container with ID starting with 89b2e0a253c56afe460daaf7c5d999c0f90ed577460e9dcbd1af94a1f5531924 not found: ID does not exist" Nov 02 22:53:41.641155 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:41.641122 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99"] Nov 02 22:53:41.694088 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:41.694048 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0efc115-559dc69989-jkw99"] Nov 02 22:53:42.145085 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:42.145048 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:42.472594 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:42.472492 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:43.313566 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:43.313533 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d865be-b81d-4cc4-82d3-3d0d4cb0b40f" path="/var/lib/kubelet/pods/41d865be-b81d-4cc4-82d3-3d0d4cb0b40f/volumes" Nov 02 22:53:43.770451 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:53:43.770416 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:53:44.957405 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:44.957367 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:53:47.383815 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:47.383764 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:48.710299 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:48.710265 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:49.232425 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.232388 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj"] Nov 02 22:53:49.232895 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.232865 5871 cpu_manager.go:414] "containerMap: removing container" podUID="41d865be-b81d-4cc4-82d3-3d0d4cb0b40f" containerName="maas-api" Nov 02 22:53:49.232953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.232896 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d865be-b81d-4cc4-82d3-3d0d4cb0b40f" containerName="maas-api" Nov 02 22:53:49.232993 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.232967 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d865be-b81d-4cc4-82d3-3d0d4cb0b40f" containerName="maas-api" Nov 02 22:53:49.236250 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.236233 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.288131 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.288094 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fed2c1-bf02-45df-8375-fda1273289e8-istio-token\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.288329 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.288138 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdb2h\" (UniqueName: \"kubernetes.io/projected/51fed2c1-bf02-45df-8375-fda1273289e8-kube-api-access-rdb2h\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.288329 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.288272 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fed2c1-bf02-45df-8375-fda1273289e8-istiod-ca-cert\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.288329 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.288318 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-credential-socket\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.288508 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.288404 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fed2c1-bf02-45df-8375-fda1273289e8-istio-podinfo\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.288508 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.288477 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-istio-data\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.288616 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.288517 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-workload-certs\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.288616 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.288543 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-istio-envoy\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.288733 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.288624 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-workload-socket\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.289972 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.289946 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-9d4413-openshift-default-dockercfg-zr2dt\"" Nov 02 22:53:49.343790 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.343750 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj"] Nov 02 22:53:49.354367 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.354337 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:49.389453 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.389418 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fed2c1-bf02-45df-8375-fda1273289e8-istio-podinfo\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.389453 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.389460 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-istio-data\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.389716 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.389485 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-workload-certs\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.389716 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.389516 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-istio-envoy\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.389716 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.389564 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-workload-socket\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.389716 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.389629 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fed2c1-bf02-45df-8375-fda1273289e8-istio-token\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.389716 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.389663 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdb2h\" (UniqueName: \"kubernetes.io/projected/51fed2c1-bf02-45df-8375-fda1273289e8-kube-api-access-rdb2h\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.390035 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.389781 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fed2c1-bf02-45df-8375-fda1273289e8-istiod-ca-cert\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.390035 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.389833 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-credential-socket\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.390035 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.389949 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-istio-data\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.390187 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.390026 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-workload-certs\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.390187 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.390156 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-workload-socket\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.390187 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.390173 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-credential-socket\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.390745 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.390718 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fed2c1-bf02-45df-8375-fda1273289e8-istiod-ca-cert\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.391979 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.391951 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-istio-envoy\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.392122 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.392103 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fed2c1-bf02-45df-8375-fda1273289e8-istio-podinfo\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.452519 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.452455 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdb2h\" (UniqueName: \"kubernetes.io/projected/51fed2c1-bf02-45df-8375-fda1273289e8-kube-api-access-rdb2h\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.455619 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.455588 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fed2c1-bf02-45df-8375-fda1273289e8-istio-token\") pod \"e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.537118 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.537003 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:49.549569 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.549537 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:49.988461 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.988435 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj"] Nov 02 22:53:49.991202 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:53:49.991173 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fed2c1_bf02_45df_8375_fda1273289e8.slice/crio-4533cc350c9a46360541860fe5ce1b645719ae056d88cecff164af421acdbea5 WatchSource:0}: Error finding container 4533cc350c9a46360541860fe5ce1b645719ae056d88cecff164af421acdbea5: Status 404 returned error can't find the container with id 4533cc350c9a46360541860fe5ce1b645719ae056d88cecff164af421acdbea5 Nov 02 22:53:49.993343 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.993315 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:53:49.993434 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.993384 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:53:49.993434 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:49.993412 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:53:50.529168 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:50.529128 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" event={"ID":"51fed2c1-bf02-45df-8375-fda1273289e8","Type":"ContainerStarted","Data":"14828f65154612723f30a8a3c0a878cdcae7e25d7d8b903bcbc37c31538f7889"} Nov 02 22:53:50.529388 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:50.529197 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" event={"ID":"51fed2c1-bf02-45df-8375-fda1273289e8","Type":"ContainerStarted","Data":"4533cc350c9a46360541860fe5ce1b645719ae056d88cecff164af421acdbea5"} Nov 02 22:53:50.550570 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:50.550532 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:50.654530 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:50.654468 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" podStartSLOduration=1.654454355 podStartE2EDuration="1.654454355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:50.65189909 +0000 UTC m=+2853.464229267" watchObservedRunningTime="2026-11-02 22:53:50.654454355 +0000 UTC m=+2853.466784532" Nov 02 22:53:51.555499 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:51.555466 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:52.539478 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:52.539447 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:52.540532 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:52.540508 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:53:53.445286 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:53.445249 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:53.642113 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:53.642080 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:53.863800 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:53.863765 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:59.608294 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:59.608262 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp"] Nov 02 22:53:59.612446 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:59.612425 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:53:59.667813 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:59.667779 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp"] Nov 02 22:53:59.692907 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:59.692850 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvxwm\" (UniqueName: \"kubernetes.io/projected/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-kube-api-access-bvxwm\") pod \"maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:53:59.693087 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:59.692916 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-metrics-certs\") pod \"maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:53:59.693087 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:59.693028 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:53:59.693087 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:59.693064 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:53:59.706212 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:59.706180 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-9d4413-serving-cert\"" Nov 02 22:53:59.793811 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:59.793769 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvxwm\" (UniqueName: \"kubernetes.io/projected/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-kube-api-access-bvxwm\") pod \"maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:53:59.794019 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:59.793816 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-metrics-certs\") pod \"maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:53:59.794019 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:59.793889 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:53:59.794019 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:59.793923 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:53:59.794170 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:53:59.794045 5871 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-serving-cert: secret "maas-api-e2e-authcfg-http-9d4413-serving-cert" not found Nov 02 22:53:59.794170 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:53:59.794136 5871 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-maas-api-tls podName:04ed1ff8-77c2-4672-95f7-b59c8a44ba84 nodeName:}" failed. No retries permitted until 2026-11-02 22:54:00.294114755 +0000 UTC m=+2863.106444933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-maas-api-tls") pod "maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" (UID: "04ed1ff8-77c2-4672-95f7-b59c8a44ba84") : secret "maas-api-e2e-authcfg-http-9d4413-serving-cert" not found Nov 02 22:53:59.794634 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:59.794610 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:53:59.796337 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:59.796314 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-metrics-certs\") pod \"maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:53:59.822456 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:59.822407 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:53:59.855683 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:53:59.855649 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvxwm\" (UniqueName: \"kubernetes.io/projected/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-kube-api-access-bvxwm\") pod \"maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:54:00.005385 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:00.005344 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:00.187741 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:00.187708 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:00.299687 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:00.299582 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:54:00.302263 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:00.302231 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:54:00.525856 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:00.525811 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:54:00.905588 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:00.905559 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp"] Nov 02 22:54:00.908263 ip-10-0-140-221 kubenswrapper[5871]: W1102 22:54:00.908229 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ed1ff8_77c2_4672_95f7_b59c8a44ba84.slice/crio-c11cb8dfaa1d02e170176618052e0b470aebf92d76cb69d463104c22b250e633 WatchSource:0}: Error finding container c11cb8dfaa1d02e170176618052e0b470aebf92d76cb69d463104c22b250e633: Status 404 returned error can't find the container with id c11cb8dfaa1d02e170176618052e0b470aebf92d76cb69d463104c22b250e633 Nov 02 22:54:01.599788 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:01.599750 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" event={"ID":"04ed1ff8-77c2-4672-95f7-b59c8a44ba84","Type":"ContainerStarted","Data":"c11cb8dfaa1d02e170176618052e0b470aebf92d76cb69d463104c22b250e633"} Nov 02 22:54:02.606699 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:02.606665 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" event={"ID":"04ed1ff8-77c2-4672-95f7-b59c8a44ba84","Type":"ContainerStarted","Data":"5082540e11d94cd2cc289fdaa853f8df86e2ad0480a826df3c374cb3e7ec4814"} Nov 02 22:54:02.607184 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:02.606755 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:54:02.726509 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:02.726452 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" podStartSLOduration=2.6094334999999997 podStartE2EDuration="3.726438677s" totalImagesPullingTime="1.117005177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:02.724567593 +0000 UTC m=+2865.536897774" watchObservedRunningTime="2026-11-02 22:54:02.726438677 +0000 UTC m=+2865.538768854" Nov 02 22:54:03.998230 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:03.998192 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:04.904624 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:04.904590 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:05.455853 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:05.455787 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:05.638867 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:05.638832 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:07.954797 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:07.954755 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:54:08.618249 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:08.618215 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:54:10.001768 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:10.001736 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:12.352083 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:12.352045 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:13.771286 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:54:13.771246 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:54:14.913967 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:14.913922 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:16.720182 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:16.720146 5871 generic.go:356] "Generic (PLEG): container finished" podID="51fed2c1-bf02-45df-8375-fda1273289e8" containerID="14828f65154612723f30a8a3c0a878cdcae7e25d7d8b903bcbc37c31538f7889" exitCode=0 Nov 02 22:54:16.720639 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:16.720204 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" event={"ID":"51fed2c1-bf02-45df-8375-fda1273289e8","Type":"ContainerDied","Data":"14828f65154612723f30a8a3c0a878cdcae7e25d7d8b903bcbc37c31538f7889"} Nov 02 22:54:16.720639 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:16.720588 5871 scope.go:122] "RemoveContainer" containerID="14828f65154612723f30a8a3c0a878cdcae7e25d7d8b903bcbc37c31538f7889" Nov 02 22:54:16.721857 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:16.721828 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:54:16.722052 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:16.722031 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:54:16.722126 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:16.722089 5871 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:54:17.728277 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:17.728242 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" event={"ID":"51fed2c1-bf02-45df-8375-fda1273289e8","Type":"ContainerStarted","Data":"111f29b10e6c25acda9a45705621f1f3e5da3ba8e1ede992cd9f418ff74f2b78"} Nov 02 22:54:18.550274 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:18.550183 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:54:18.551588 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:18.551555 5871 patch_prober.go:29] interesting pod/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 22:54:18.551723 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:18.551607 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" podUID="51fed2c1-bf02-45df-8375-fda1273289e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 22:54:19.550361 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:19.550315 5871 patch_prober.go:29] interesting pod/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 22:54:19.550828 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:19.550386 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" podUID="51fed2c1-bf02-45df-8375-fda1273289e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 22:54:20.199640 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:20.199602 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:20.550724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:20.550639 5871 patch_prober.go:29] interesting pod/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 22:54:20.550724 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:20.550705 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" podUID="51fed2c1-bf02-45df-8375-fda1273289e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 22:54:21.550287 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:21.550249 5871 patch_prober.go:29] interesting pod/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 22:54:21.550485 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:21.550322 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" podUID="51fed2c1-bf02-45df-8375-fda1273289e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 22:54:22.539831 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:22.539771 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:54:22.550685 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:22.550653 5871 patch_prober.go:29] interesting pod/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 22:54:22.550850 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:22.550705 5871 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" podUID="51fed2c1-bf02-45df-8375-fda1273289e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 22:54:23.555110 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:23.555071 5871 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:54:23.556088 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:23.556068 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:54:25.223681 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:25.223592 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:28.363691 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:28.363647 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:30.205100 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:30.205065 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:33.596255 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:33.596216 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:35.200418 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:35.200384 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:40.194020 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:40.193954 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:43.770477 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:54:43.770435 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:54:45.095764 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:45.095725 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:50.089378 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:50.089343 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:54.993213 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:54.993180 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:54:59.987000 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:54:59.986965 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:04.348861 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:55:04.348824 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:04.895529 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:55:04.895493 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:09.894297 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:55:09.894249 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:12.681319 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:55:12.681283 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:13.771937 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:55:13.771900 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:55:14.883139 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:55:14.883102 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:19.893153 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:55:19.893113 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:24.790286 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:55:24.790254 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:29.744754 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:55:29.744722 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:33.927662 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:55:33.927625 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:34.826466 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:55:34.826409 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:39.808028 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:55:39.807991 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:43.769442 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:55:43.769400 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:55:44.800059 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:55:44.800021 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:49.794081 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:55:49.794045 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:54.707425 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:55:54.707341 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:55:59.939263 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:55:59.939221 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:04.670168 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:04.670130 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:05.005794 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:05.005710 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:10.701772 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:10.701727 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:10.884133 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:10.884101 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:13.035314 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:13.035278 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:13.770111 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:56:13.770072 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:56:19.122392 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.122356 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp"] Nov 02 22:56:19.122886 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.122566 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" podUID="04ed1ff8-77c2-4672-95f7-b59c8a44ba84" containerName="maas-api" containerID="cri-o://5082540e11d94cd2cc289fdaa853f8df86e2ad0480a826df3c374cb3e7ec4814" gracePeriod=30 Nov 02 22:56:19.255159 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.255093 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:19.378413 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.378344 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:56:19.405415 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.405378 5871 generic.go:356] "Generic (PLEG): container finished" podID="04ed1ff8-77c2-4672-95f7-b59c8a44ba84" containerID="5082540e11d94cd2cc289fdaa853f8df86e2ad0480a826df3c374cb3e7ec4814" exitCode=0 Nov 02 22:56:19.405622 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.405425 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" Nov 02 22:56:19.405622 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.405466 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" event={"ID":"04ed1ff8-77c2-4672-95f7-b59c8a44ba84","Type":"ContainerDied","Data":"5082540e11d94cd2cc289fdaa853f8df86e2ad0480a826df3c374cb3e7ec4814"} Nov 02 22:56:19.405622 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.405528 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp" event={"ID":"04ed1ff8-77c2-4672-95f7-b59c8a44ba84","Type":"ContainerDied","Data":"c11cb8dfaa1d02e170176618052e0b470aebf92d76cb69d463104c22b250e633"} Nov 02 22:56:19.405622 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.405561 5871 scope.go:122] "RemoveContainer" containerID="5082540e11d94cd2cc289fdaa853f8df86e2ad0480a826df3c374cb3e7ec4814" Nov 02 22:56:19.415121 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.415095 5871 scope.go:122] "RemoveContainer" containerID="5082540e11d94cd2cc289fdaa853f8df86e2ad0480a826df3c374cb3e7ec4814" Nov 02 22:56:19.415451 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:56:19.415429 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5082540e11d94cd2cc289fdaa853f8df86e2ad0480a826df3c374cb3e7ec4814\": container with ID starting with 5082540e11d94cd2cc289fdaa853f8df86e2ad0480a826df3c374cb3e7ec4814 not found: ID does not exist" containerID="5082540e11d94cd2cc289fdaa853f8df86e2ad0480a826df3c374cb3e7ec4814" Nov 02 22:56:19.415527 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.415466 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5082540e11d94cd2cc289fdaa853f8df86e2ad0480a826df3c374cb3e7ec4814"} err="failed to get container status \"5082540e11d94cd2cc289fdaa853f8df86e2ad0480a826df3c374cb3e7ec4814\": rpc error: code = NotFound desc = could not find container \"5082540e11d94cd2cc289fdaa853f8df86e2ad0480a826df3c374cb3e7ec4814\": container with ID starting with 5082540e11d94cd2cc289fdaa853f8df86e2ad0480a826df3c374cb3e7ec4814 not found: ID does not exist" Nov 02 22:56:19.429984 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.429941 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-metrics-certs\") pod \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " Nov 02 22:56:19.430170 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.430008 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-kube-api-access-bvxwm\" (UniqueName: \"kubernetes.io/projected/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-kube-api-access-bvxwm\") pod \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " Nov 02 22:56:19.430170 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.430082 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-maas-api-tls\") pod \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " Nov 02 22:56:19.430301 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.430188 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-openshift-service-ca\") pod \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\" (UID: \"04ed1ff8-77c2-4672-95f7-b59c8a44ba84\") " Nov 02 22:56:19.430563 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.430534 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-openshift-service-ca" pod "04ed1ff8-77c2-4672-95f7-b59c8a44ba84" (UID: "04ed1ff8-77c2-4672-95f7-b59c8a44ba84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:56:19.432237 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.432205 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-metrics-certs" pod "04ed1ff8-77c2-4672-95f7-b59c8a44ba84" (UID: "04ed1ff8-77c2-4672-95f7-b59c8a44ba84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:56:19.432363 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.432266 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-kube-api-access-bvxwm" pod "04ed1ff8-77c2-4672-95f7-b59c8a44ba84" (UID: "04ed1ff8-77c2-4672-95f7-b59c8a44ba84"). InnerVolumeSpecName "kube-api-access-bvxwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:19.432363 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.432297 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-maas-api-tls" pod "04ed1ff8-77c2-4672-95f7-b59c8a44ba84" (UID: "04ed1ff8-77c2-4672-95f7-b59c8a44ba84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:56:19.532132 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.532091 5871 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-maas-api-tls\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:19.532132 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.532126 5871 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-openshift-service-ca\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:19.532349 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.532136 5871 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-metrics-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:19.532349 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.532160 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvxwm\" (UniqueName: \"kubernetes.io/projected/04ed1ff8-77c2-4672-95f7-b59c8a44ba84-kube-api-access-bvxwm\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:19.828609 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.828571 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp"] Nov 02 22:56:19.833789 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:19.833758 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-9d4413-849844dbc7-q2csp"] Nov 02 22:56:20.828353 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:20.828313 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:21.122206 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:21.122169 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:21.313846 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:21.313813 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04ed1ff8-77c2-4672-95f7-b59c8a44ba84" path="/var/lib/kubelet/pods/04ed1ff8-77c2-4672-95f7-b59c8a44ba84/volumes" Nov 02 22:56:21.910994 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:21.910961 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:23.770863 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:23.770824 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:56:28.221107 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:28.221071 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:28.404483 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:28.404448 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:28.546201 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:28.546118 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj"] Nov 02 22:56:28.546367 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:28.546346 5871 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" podUID="51fed2c1-bf02-45df-8375-fda1273289e8" containerName="istio-proxy" containerID="cri-o://111f29b10e6c25acda9a45705621f1f3e5da3ba8e1ede992cd9f418ff74f2b78" gracePeriod=30 Nov 02 22:56:28.582006 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:28.581937 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:33.805137 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.805110 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:56:33.885378 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.885341 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51fed2c1-bf02-45df-8375-fda1273289e8-istio-token\" (UniqueName: \"kubernetes.io/projected/51fed2c1-bf02-45df-8375-fda1273289e8-istio-token\") pod \"51fed2c1-bf02-45df-8375-fda1273289e8\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " Nov 02 22:56:33.885605 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.885426 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51fed2c1-bf02-45df-8375-fda1273289e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fed2c1-bf02-45df-8375-fda1273289e8-istiod-ca-cert\") pod \"51fed2c1-bf02-45df-8375-fda1273289e8\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " Nov 02 22:56:33.885605 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.885531 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-workload-socket\") pod \"51fed2c1-bf02-45df-8375-fda1273289e8\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " Nov 02 22:56:33.885605 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.885567 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51fed2c1-bf02-45df-8375-fda1273289e8-kube-api-access-rdb2h\" (UniqueName: \"kubernetes.io/projected/51fed2c1-bf02-45df-8375-fda1273289e8-kube-api-access-rdb2h\") pod \"51fed2c1-bf02-45df-8375-fda1273289e8\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " Nov 02 22:56:33.885775 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.885621 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51fed2c1-bf02-45df-8375-fda1273289e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fed2c1-bf02-45df-8375-fda1273289e8-istio-podinfo\") pod \"51fed2c1-bf02-45df-8375-fda1273289e8\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " Nov 02 22:56:33.885775 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.885656 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-istio-envoy\") pod \"51fed2c1-bf02-45df-8375-fda1273289e8\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " Nov 02 22:56:33.885775 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.885703 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-credential-socket\") pod \"51fed2c1-bf02-45df-8375-fda1273289e8\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " Nov 02 22:56:33.885775 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.885749 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-workload-certs\") pod \"51fed2c1-bf02-45df-8375-fda1273289e8\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " Nov 02 22:56:33.886005 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.885823 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-workload-socket" pod "51fed2c1-bf02-45df-8375-fda1273289e8" (UID: "51fed2c1-bf02-45df-8375-fda1273289e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:33.886005 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.885843 5871 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-istio-data\") pod \"51fed2c1-bf02-45df-8375-fda1273289e8\" (UID: \"51fed2c1-bf02-45df-8375-fda1273289e8\") " Nov 02 22:56:33.886200 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.885841 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fed2c1-bf02-45df-8375-fda1273289e8-istiod-ca-cert" pod "51fed2c1-bf02-45df-8375-fda1273289e8" (UID: "51fed2c1-bf02-45df-8375-fda1273289e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:56:33.886269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.886101 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-credential-socket" pod "51fed2c1-bf02-45df-8375-fda1273289e8" (UID: "51fed2c1-bf02-45df-8375-fda1273289e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:33.886269 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.886110 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-workload-certs" pod "51fed2c1-bf02-45df-8375-fda1273289e8" (UID: "51fed2c1-bf02-45df-8375-fda1273289e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:33.886380 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.886287 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-istio-data" pod "51fed2c1-bf02-45df-8375-fda1273289e8" (UID: "51fed2c1-bf02-45df-8375-fda1273289e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:33.886380 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.886307 5871 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fed2c1-bf02-45df-8375-fda1273289e8-istiod-ca-cert\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:33.886380 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.886322 5871 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-workload-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:33.886380 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.886332 5871 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-credential-socket\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:33.886380 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.886343 5871 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-workload-certs\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:33.888007 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.887984 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fed2c1-bf02-45df-8375-fda1273289e8-kube-api-access-rdb2h" pod "51fed2c1-bf02-45df-8375-fda1273289e8" (UID: "51fed2c1-bf02-45df-8375-fda1273289e8"). InnerVolumeSpecName "kube-api-access-rdb2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:33.888198 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.888179 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fed2c1-bf02-45df-8375-fda1273289e8-istio-token" pod "51fed2c1-bf02-45df-8375-fda1273289e8" (UID: "51fed2c1-bf02-45df-8375-fda1273289e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:56:33.888271 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.888190 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51fed2c1-bf02-45df-8375-fda1273289e8-istio-podinfo" pod "51fed2c1-bf02-45df-8375-fda1273289e8" (UID: "51fed2c1-bf02-45df-8375-fda1273289e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:56:33.888314 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.888262 5871 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-istio-envoy" pod "51fed2c1-bf02-45df-8375-fda1273289e8" (UID: "51fed2c1-bf02-45df-8375-fda1273289e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:56:33.987889 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.987841 5871 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-istio-data\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:33.987889 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.987895 5871 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fed2c1-bf02-45df-8375-fda1273289e8-istio-token\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:33.988166 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.987908 5871 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdb2h\" (UniqueName: \"kubernetes.io/projected/51fed2c1-bf02-45df-8375-fda1273289e8-kube-api-access-rdb2h\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:33.988166 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.987918 5871 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fed2c1-bf02-45df-8375-fda1273289e8-istio-podinfo\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:33.988166 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:33.987928 5871 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fed2c1-bf02-45df-8375-fda1273289e8-istio-envoy\") on node \"ip-10-0-140-221.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:56:34.270749 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:34.270640 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:34.500288 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:34.500254 5871 generic.go:356] "Generic (PLEG): container finished" podID="51fed2c1-bf02-45df-8375-fda1273289e8" containerID="111f29b10e6c25acda9a45705621f1f3e5da3ba8e1ede992cd9f418ff74f2b78" exitCode=0 Nov 02 22:56:34.500497 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:34.500315 5871 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" Nov 02 22:56:34.500497 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:34.500337 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" event={"ID":"51fed2c1-bf02-45df-8375-fda1273289e8","Type":"ContainerDied","Data":"111f29b10e6c25acda9a45705621f1f3e5da3ba8e1ede992cd9f418ff74f2b78"} Nov 02 22:56:34.500497 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:34.500397 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj" event={"ID":"51fed2c1-bf02-45df-8375-fda1273289e8","Type":"ContainerDied","Data":"4533cc350c9a46360541860fe5ce1b645719ae056d88cecff164af421acdbea5"} Nov 02 22:56:34.500497 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:34.500419 5871 scope.go:122] "RemoveContainer" containerID="111f29b10e6c25acda9a45705621f1f3e5da3ba8e1ede992cd9f418ff74f2b78" Nov 02 22:56:34.510417 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:34.510395 5871 scope.go:122] "RemoveContainer" containerID="14828f65154612723f30a8a3c0a878cdcae7e25d7d8b903bcbc37c31538f7889" Nov 02 22:56:34.519395 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:34.519375 5871 scope.go:122] "RemoveContainer" containerID="111f29b10e6c25acda9a45705621f1f3e5da3ba8e1ede992cd9f418ff74f2b78" Nov 02 22:56:34.519710 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:56:34.519689 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111f29b10e6c25acda9a45705621f1f3e5da3ba8e1ede992cd9f418ff74f2b78\": container with ID starting with 111f29b10e6c25acda9a45705621f1f3e5da3ba8e1ede992cd9f418ff74f2b78 not found: ID does not exist" containerID="111f29b10e6c25acda9a45705621f1f3e5da3ba8e1ede992cd9f418ff74f2b78" Nov 02 22:56:34.519759 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:34.519721 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111f29b10e6c25acda9a45705621f1f3e5da3ba8e1ede992cd9f418ff74f2b78"} err="failed to get container status \"111f29b10e6c25acda9a45705621f1f3e5da3ba8e1ede992cd9f418ff74f2b78\": rpc error: code = NotFound desc = could not find container \"111f29b10e6c25acda9a45705621f1f3e5da3ba8e1ede992cd9f418ff74f2b78\": container with ID starting with 111f29b10e6c25acda9a45705621f1f3e5da3ba8e1ede992cd9f418ff74f2b78 not found: ID does not exist" Nov 02 22:56:34.519759 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:34.519740 5871 scope.go:122] "RemoveContainer" containerID="14828f65154612723f30a8a3c0a878cdcae7e25d7d8b903bcbc37c31538f7889" Nov 02 22:56:34.520020 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:56:34.520001 5871 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14828f65154612723f30a8a3c0a878cdcae7e25d7d8b903bcbc37c31538f7889\": container with ID starting with 14828f65154612723f30a8a3c0a878cdcae7e25d7d8b903bcbc37c31538f7889 not found: ID does not exist" containerID="14828f65154612723f30a8a3c0a878cdcae7e25d7d8b903bcbc37c31538f7889" Nov 02 22:56:34.520083 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:34.520030 5871 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14828f65154612723f30a8a3c0a878cdcae7e25d7d8b903bcbc37c31538f7889"} err="failed to get container status \"14828f65154612723f30a8a3c0a878cdcae7e25d7d8b903bcbc37c31538f7889\": rpc error: code = NotFound desc = could not find container \"14828f65154612723f30a8a3c0a878cdcae7e25d7d8b903bcbc37c31538f7889\": container with ID starting with 14828f65154612723f30a8a3c0a878cdcae7e25d7d8b903bcbc37c31538f7889 not found: ID does not exist" Nov 02 22:56:34.667459 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:34.667427 5871 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj"] Nov 02 22:56:34.717308 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:34.717274 5871 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-9d4413-openshift-default-555fdf46cc-8kcjj"] Nov 02 22:56:35.313143 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:35.313109 5871 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51fed2c1-bf02-45df-8375-fda1273289e8" path="/var/lib/kubelet/pods/51fed2c1-bf02-45df-8375-fda1273289e8/volumes" Nov 02 22:56:36.540459 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:36.540421 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:56:43.769618 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:56:43.769578 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:56:47.847152 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:47.847111 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:56:48.041194 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:48.041145 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:56:48.224623 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:56:48.224584 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:01.191960 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:57:01.191924 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:01.375212 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:57:01.375171 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:05.070407 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:57:05.070368 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:09.790423 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:57:09.790387 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:09.972086 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:57:09.972050 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:13.381293 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:57:13.381252 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:13.770210 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:57:13.770119 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:57:18.861633 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:57:18.861595 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:57:34.625320 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:57:34.625233 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:57:43.769349 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:57:43.769309 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:58:00.035343 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:00.035305 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gzdhm"] Nov 02 22:58:05.364031 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:05.363974 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:13.711917 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:13.711857 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:13.770609 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:58:13.770569 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:58:19.851227 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:19.851193 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54d8d5fc5c-6k894_313ca49b-f935-4457-a2a3-afd36e3477e5/manager/0.log" Nov 02 22:58:22.040864 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:22.040829 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mn4sc_28482f90-219d-4eae-b413-f5413c9ef08b/manager/0.log" Nov 02 22:58:22.295413 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:22.295310 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kqst9_954286ee-b7ba-4a67-9c30-904f75d92433/proxy/0.log" Nov 02 22:58:23.373326 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:23.373291 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-86c65979d7-v94d5_a66b293a-0f29-4692-97d7-c64412b84758/maas-api/0.log" Nov 02 22:58:26.300539 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:26.300507 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-p9229_aff64e43-f4d1-4e2e-927a-c83ad7861193/manager/0.log" Nov 02 22:58:27.453184 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:27.453149 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gzdhm_6f279b88-919c-4c3d-a46c-f5af9b1fbd63/limitador/0.log" Nov 02 22:58:27.675396 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:27.675367 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-scqp5_fb6fface-d61e-4b8b-8a0a-783123e5c1fe/manager/0.log" Nov 02 22:58:29.156853 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:29.156818 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f985b8d7-l5mgd_cd2067a6-51bb-4d34-ae53-cfd425b26c3b/kube-auth-proxy/0.log" Nov 02 22:58:30.718717 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:30.718680 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-jh4w5_440d8a6a-030e-4aad-a190-85fc0292617b/istio-proxy/0.log" Nov 02 22:58:31.456682 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:31.456648 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-pf98d_8fd7034d-1b03-47bb-bd55-53bbd67ec740/istio-proxy/0.log" Nov 02 22:58:31.732677 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:31.732592 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-tfgpj_76dcefe6-1e67-4ba4-98a6-9358212dee30/istio-proxy/0.log" Nov 02 22:58:34.027225 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:34.027184 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-tmtfl_57e8906d-5f46-4056-baa4-a1401866e0ad/main/0.log" Nov 02 22:58:34.509370 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:34.509343 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-6sz2m_5320d06b-d052-42e4-a0d1-be48e14ba7ee/main/0.log" Nov 02 22:58:34.958840 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:34.958787 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8qtld_a4bfaefa-f570-4bf9-abd6-45231654dafe/main/0.log" Nov 02 22:58:35.419674 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:35.419644 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7_ccce4075-668d-487a-8a75-fbd41fee023a/main/0.log" Nov 02 22:58:36.882223 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:36.882186 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:43.771227 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:58:43.771185 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:58:44.676708 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:44.676668 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:58:55.693323 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:55.693228 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54d8d5fc5c-6k894_313ca49b-f935-4457-a2a3-afd36e3477e5/manager/0.log" Nov 02 22:58:57.458516 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:57.458476 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mn4sc_28482f90-219d-4eae-b413-f5413c9ef08b/manager/0.log" Nov 02 22:58:57.716669 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:57.716588 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kqst9_954286ee-b7ba-4a67-9c30-904f75d92433/proxy/0.log" Nov 02 22:58:58.587665 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:58:58.587635 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-86c65979d7-v94d5_a66b293a-0f29-4692-97d7-c64412b84758/maas-api/0.log" Nov 02 22:59:01.089917 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:59:01.089866 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-p9229_aff64e43-f4d1-4e2e-927a-c83ad7861193/manager/0.log" Nov 02 22:59:02.231072 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:59:02.231032 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gzdhm_6f279b88-919c-4c3d-a46c-f5af9b1fbd63/limitador/0.log" Nov 02 22:59:02.457351 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:59:02.457304 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-scqp5_fb6fface-d61e-4b8b-8a0a-783123e5c1fe/manager/0.log" Nov 02 22:59:03.702574 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:59:03.702544 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f985b8d7-l5mgd_cd2067a6-51bb-4d34-ae53-cfd425b26c3b/kube-auth-proxy/0.log" Nov 02 22:59:05.709750 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:59:05.709706 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:59:13.770558 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:59:13.770514 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 22:59:14.072294 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:59:14.072212 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:59:15.337911 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:59:15.337857 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-jh4w5_440d8a6a-030e-4aad-a190-85fc0292617b/istio-proxy/0.log" Nov 02 22:59:16.137774 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:59:16.137720 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-pf98d_8fd7034d-1b03-47bb-bd55-53bbd67ec740/istio-proxy/0.log" Nov 02 22:59:16.463241 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:59:16.463158 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-tfgpj_76dcefe6-1e67-4ba4-98a6-9358212dee30/istio-proxy/0.log" Nov 02 22:59:18.544570 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:59:18.544542 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-tmtfl_57e8906d-5f46-4056-baa4-a1401866e0ad/main/0.log" Nov 02 22:59:18.987953 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:59:18.987924 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-6sz2m_5320d06b-d052-42e4-a0d1-be48e14ba7ee/main/0.log" Nov 02 22:59:19.434830 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:59:19.434799 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8qtld_a4bfaefa-f570-4bf9-abd6-45231654dafe/main/0.log" Nov 02 22:59:19.895747 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:59:19.895716 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-2krg7_ccce4075-668d-487a-8a75-fbd41fee023a/main/0.log" Nov 02 22:59:35.325041 ip-10-0-140-221 kubenswrapper[5871]: I1102 22:59:35.325007 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 22:59:43.770427 ip-10-0-140-221 kubenswrapper[5871]: E1102 22:59:43.770388 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 23:00:06.071622 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:00:06.071583 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:13.771342 ip-10-0-140-221 kubenswrapper[5871]: E1102 23:00:13.771302 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 23:00:14.411061 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:00:14.411024 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 23:00:43.769604 ip-10-0-140-221 kubenswrapper[5871]: E1102 23:00:43.769512 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 23:00:45.688504 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:00:45.688464 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:06.358045 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:06.358004 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:08.560071 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:08.560034 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:13.770970 ip-10-0-140-221 kubenswrapper[5871]: E1102 23:01:13.770930 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 23:01:14.748166 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:14.748130 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:29.376003 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:29.375964 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:31.257420 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:31.257383 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:31.487719 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:31.487688 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:43.774056 ip-10-0-140-221 kubenswrapper[5871]: E1102 23:01:43.774012 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 23:01:43.831087 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.831052 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-drfxk/must-gather-wxpbn"] Nov 02 23:01:43.831504 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.831491 5871 cpu_manager.go:414] "containerMap: removing container" podUID="51fed2c1-bf02-45df-8375-fda1273289e8" containerName="istio-proxy" Nov 02 23:01:43.831549 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.831507 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51fed2c1-bf02-45df-8375-fda1273289e8" containerName="istio-proxy" Nov 02 23:01:43.831549 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.831522 5871 cpu_manager.go:414] "containerMap: removing container" podUID="51fed2c1-bf02-45df-8375-fda1273289e8" containerName="istio-proxy" Nov 02 23:01:43.831549 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.831527 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51fed2c1-bf02-45df-8375-fda1273289e8" containerName="istio-proxy" Nov 02 23:01:43.831642 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.831555 5871 cpu_manager.go:414] "containerMap: removing container" podUID="04ed1ff8-77c2-4672-95f7-b59c8a44ba84" containerName="maas-api" Nov 02 23:01:43.831642 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.831562 5871 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04ed1ff8-77c2-4672-95f7-b59c8a44ba84" containerName="maas-api" Nov 02 23:01:43.831707 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.831644 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="04ed1ff8-77c2-4672-95f7-b59c8a44ba84" containerName="maas-api" Nov 02 23:01:43.831707 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.831657 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="51fed2c1-bf02-45df-8375-fda1273289e8" containerName="istio-proxy" Nov 02 23:01:43.831707 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.831664 5871 memory_manager.go:365] "RemoveStaleState removing state" podUID="51fed2c1-bf02-45df-8375-fda1273289e8" containerName="istio-proxy" Nov 02 23:01:43.835245 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.835223 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-drfxk/must-gather-wxpbn" Nov 02 23:01:43.852343 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.852311 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:43.888169 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.888135 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-drfxk/must-gather-wxpbn"] Nov 02 23:01:43.931736 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.931689 5871 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-drfxk\"/\"default-dockercfg-x5w7p\"" Nov 02 23:01:43.931736 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.931701 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-drfxk\"/\"openshift-service-ca.crt\"" Nov 02 23:01:43.931985 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.931722 5871 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-drfxk\"/\"kube-root-ca.crt\"" Nov 02 23:01:43.950819 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.950777 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nsf8\" (UniqueName: \"kubernetes.io/projected/fb6b72a3-a3cc-4abd-b797-23143e8bd2d4-kube-api-access-7nsf8\") pod \"must-gather-wxpbn\" (UID: \"fb6b72a3-a3cc-4abd-b797-23143e8bd2d4\") " pod="openshift-must-gather-drfxk/must-gather-wxpbn" Nov 02 23:01:43.951026 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:43.950924 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fb6b72a3-a3cc-4abd-b797-23143e8bd2d4-must-gather-output\") pod \"must-gather-wxpbn\" (UID: \"fb6b72a3-a3cc-4abd-b797-23143e8bd2d4\") " pod="openshift-must-gather-drfxk/must-gather-wxpbn" Nov 02 23:01:44.035493 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:44.035410 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:44.052378 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:44.052339 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nsf8\" (UniqueName: \"kubernetes.io/projected/fb6b72a3-a3cc-4abd-b797-23143e8bd2d4-kube-api-access-7nsf8\") pod \"must-gather-wxpbn\" (UID: \"fb6b72a3-a3cc-4abd-b797-23143e8bd2d4\") " pod="openshift-must-gather-drfxk/must-gather-wxpbn" Nov 02 23:01:44.052560 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:44.052418 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fb6b72a3-a3cc-4abd-b797-23143e8bd2d4-must-gather-output\") pod \"must-gather-wxpbn\" (UID: \"fb6b72a3-a3cc-4abd-b797-23143e8bd2d4\") " pod="openshift-must-gather-drfxk/must-gather-wxpbn" Nov 02 23:01:44.052791 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:44.052771 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fb6b72a3-a3cc-4abd-b797-23143e8bd2d4-must-gather-output\") pod \"must-gather-wxpbn\" (UID: \"fb6b72a3-a3cc-4abd-b797-23143e8bd2d4\") " pod="openshift-must-gather-drfxk/must-gather-wxpbn" Nov 02 23:01:44.109781 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:44.109745 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nsf8\" (UniqueName: \"kubernetes.io/projected/fb6b72a3-a3cc-4abd-b797-23143e8bd2d4-kube-api-access-7nsf8\") pod \"must-gather-wxpbn\" (UID: \"fb6b72a3-a3cc-4abd-b797-23143e8bd2d4\") " pod="openshift-must-gather-drfxk/must-gather-wxpbn" Nov 02 23:01:44.149988 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:44.149945 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-drfxk/must-gather-wxpbn" Nov 02 23:01:44.525308 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:44.525265 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-drfxk/must-gather-wxpbn"] Nov 02 23:01:44.532025 ip-10-0-140-221 kubenswrapper[5871]: W1102 23:01:44.531982 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb6b72a3_a3cc_4abd_b797_23143e8bd2d4.slice/crio-2e1b979e4a0ec9ea4bc3707aa7eeeef3fae79b128610d744248845e1c9bbb8a9 WatchSource:0}: Error finding container 2e1b979e4a0ec9ea4bc3707aa7eeeef3fae79b128610d744248845e1c9bbb8a9: Status 404 returned error can't find the container with id 2e1b979e4a0ec9ea4bc3707aa7eeeef3fae79b128610d744248845e1c9bbb8a9 Nov 02 23:01:44.534000 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:44.533983 5871 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 23:01:45.264362 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:45.264326 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-drfxk/must-gather-wxpbn" event={"ID":"fb6b72a3-a3cc-4abd-b797-23143e8bd2d4","Type":"ContainerStarted","Data":"2e1b979e4a0ec9ea4bc3707aa7eeeef3fae79b128610d744248845e1c9bbb8a9"} Nov 02 23:01:46.009468 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:46.009415 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 23:01:46.273000 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:46.272911 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-drfxk/must-gather-wxpbn" event={"ID":"fb6b72a3-a3cc-4abd-b797-23143e8bd2d4","Type":"ContainerStarted","Data":"97a095749da14c7958c636d60493fee1a1b1a8bb54c7dbe7c6774a2bb5ab0108"} Nov 02 23:01:46.273000 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:46.272958 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-drfxk/must-gather-wxpbn" event={"ID":"fb6b72a3-a3cc-4abd-b797-23143e8bd2d4","Type":"ContainerStarted","Data":"ab67fb80733c55fb39d351f2d21145f094269517f15731c76b5df85cd561cdce"} Nov 02 23:01:46.383237 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:46.383178 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-drfxk/must-gather-wxpbn" podStartSLOduration=1.9446953420000002 podStartE2EDuration="3.383162971s" totalImagesPullingTime="1.438467629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:46.380984729 +0000 UTC m=+3329.193314909" watchObservedRunningTime="2026-11-02 23:01:46.383162971 +0000 UTC m=+3329.195493149" Nov 02 23:01:53.809893 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:53.806759 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tp2df_644947bd-1d54-4a7f-afb6-463500cf17b9/global-pull-secret-syncer/0.log" Nov 02 23:01:55.692766 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:55.688069 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" Nov 02 23:01:57.456019 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:01:57.455988 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-221.us-west-2.compute.internal_8332e3a50281a723271a56acbbdc782f/haproxy/0.log" Nov 02 23:02:02.954885 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:02.954835 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-p9229_aff64e43-f4d1-4e2e-927a-c83ad7861193/manager/0.log" Nov 02 23:02:03.952015 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:03.951968 5871 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl"] Nov 02 23:02:03.958310 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:03.958273 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.002838 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.002345 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df485aae-4fac-4ca7-8039-9ee7cdde99f9-sys\") pod \"perf-node-gather-daemonset-pgdrl\" (UID: \"df485aae-4fac-4ca7-8039-9ee7cdde99f9\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.002838 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.002388 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df485aae-4fac-4ca7-8039-9ee7cdde99f9-podres\") pod \"perf-node-gather-daemonset-pgdrl\" (UID: \"df485aae-4fac-4ca7-8039-9ee7cdde99f9\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.002838 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.002531 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqpjn\" (UniqueName: \"kubernetes.io/projected/df485aae-4fac-4ca7-8039-9ee7cdde99f9-kube-api-access-jqpjn\") pod \"perf-node-gather-daemonset-pgdrl\" (UID: \"df485aae-4fac-4ca7-8039-9ee7cdde99f9\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.002838 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.002579 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df485aae-4fac-4ca7-8039-9ee7cdde99f9-lib-modules\") pod \"perf-node-gather-daemonset-pgdrl\" (UID: \"df485aae-4fac-4ca7-8039-9ee7cdde99f9\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.002838 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.002687 5871 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df485aae-4fac-4ca7-8039-9ee7cdde99f9-proc\") pod \"perf-node-gather-daemonset-pgdrl\" (UID: \"df485aae-4fac-4ca7-8039-9ee7cdde99f9\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.037810 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.037774 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl"] Nov 02 23:02:04.104279 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.104235 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqpjn\" (UniqueName: \"kubernetes.io/projected/df485aae-4fac-4ca7-8039-9ee7cdde99f9-kube-api-access-jqpjn\") pod \"perf-node-gather-daemonset-pgdrl\" (UID: \"df485aae-4fac-4ca7-8039-9ee7cdde99f9\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.104279 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.104286 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df485aae-4fac-4ca7-8039-9ee7cdde99f9-lib-modules\") pod \"perf-node-gather-daemonset-pgdrl\" (UID: \"df485aae-4fac-4ca7-8039-9ee7cdde99f9\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.104592 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.104312 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df485aae-4fac-4ca7-8039-9ee7cdde99f9-proc\") pod \"perf-node-gather-daemonset-pgdrl\" (UID: \"df485aae-4fac-4ca7-8039-9ee7cdde99f9\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.104592 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.104413 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df485aae-4fac-4ca7-8039-9ee7cdde99f9-sys\") pod \"perf-node-gather-daemonset-pgdrl\" (UID: \"df485aae-4fac-4ca7-8039-9ee7cdde99f9\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.104592 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.104442 5871 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df485aae-4fac-4ca7-8039-9ee7cdde99f9-podres\") pod \"perf-node-gather-daemonset-pgdrl\" (UID: \"df485aae-4fac-4ca7-8039-9ee7cdde99f9\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.104592 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.104443 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df485aae-4fac-4ca7-8039-9ee7cdde99f9-lib-modules\") pod \"perf-node-gather-daemonset-pgdrl\" (UID: \"df485aae-4fac-4ca7-8039-9ee7cdde99f9\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.104592 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.104520 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df485aae-4fac-4ca7-8039-9ee7cdde99f9-proc\") pod \"perf-node-gather-daemonset-pgdrl\" (UID: \"df485aae-4fac-4ca7-8039-9ee7cdde99f9\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.104592 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.104552 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df485aae-4fac-4ca7-8039-9ee7cdde99f9-sys\") pod \"perf-node-gather-daemonset-pgdrl\" (UID: \"df485aae-4fac-4ca7-8039-9ee7cdde99f9\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.104592 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.104574 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df485aae-4fac-4ca7-8039-9ee7cdde99f9-podres\") pod \"perf-node-gather-daemonset-pgdrl\" (UID: \"df485aae-4fac-4ca7-8039-9ee7cdde99f9\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.164709 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.164676 5871 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqpjn\" (UniqueName: \"kubernetes.io/projected/df485aae-4fac-4ca7-8039-9ee7cdde99f9-kube-api-access-jqpjn\") pod \"perf-node-gather-daemonset-pgdrl\" (UID: \"df485aae-4fac-4ca7-8039-9ee7cdde99f9\") " pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.274138 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.274056 5871 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:04.646751 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.646720 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gzdhm_6f279b88-919c-4c3d-a46c-f5af9b1fbd63/limitador/0.log" Nov 02 23:02:04.655374 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:04.655337 5871 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl"] Nov 02 23:02:04.661781 ip-10-0-140-221 kubenswrapper[5871]: W1102 23:02:04.661741 5871 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddf485aae_4fac_4ca7_8039_9ee7cdde99f9.slice/crio-23731615e5726e38ccd8d6b1865e7b9f1eb0b717e78bfd3b68bd5907e24061d8 WatchSource:0}: Error finding container 23731615e5726e38ccd8d6b1865e7b9f1eb0b717e78bfd3b68bd5907e24061d8: Status 404 returned error can't find the container with id 23731615e5726e38ccd8d6b1865e7b9f1eb0b717e78bfd3b68bd5907e24061d8 Nov 02 23:02:05.093908 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:05.093787 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-scqp5_fb6fface-d61e-4b8b-8a0a-783123e5c1fe/manager/0.log" Nov 02 23:02:05.446512 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:05.446479 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" event={"ID":"df485aae-4fac-4ca7-8039-9ee7cdde99f9","Type":"ContainerStarted","Data":"ee3c6406e68f3ff2100123f8d9def3bb8dd1817ef8f7bb13ab043c3be00a611f"} Nov 02 23:02:05.446713 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:05.446533 5871 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" event={"ID":"df485aae-4fac-4ca7-8039-9ee7cdde99f9","Type":"ContainerStarted","Data":"23731615e5726e38ccd8d6b1865e7b9f1eb0b717e78bfd3b68bd5907e24061d8"} Nov 02 23:02:05.446713 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:05.446639 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:05.559437 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:05.559380 5871 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" podStartSLOduration=2.559363703 podStartE2EDuration="2.559363703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 23:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:05.558276593 +0000 UTC m=+3348.370606770" watchObservedRunningTime="2026-11-02 23:02:05.559363703 +0000 UTC m=+3348.371693893" Nov 02 23:02:06.739581 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:06.739544 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:11.249417 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:11.249385 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-g729w_1f3ab9d6-1afa-499c-80d2-a1d11981eeb2/kube-state-metrics/0.log" Nov 02 23:02:11.462832 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:11.462799 5871 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-drfxk/perf-node-gather-daemonset-pgdrl" Nov 02 23:02:11.660315 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:11.660284 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-g729w_1f3ab9d6-1afa-499c-80d2-a1d11981eeb2/kube-rbac-proxy-main/0.log" Nov 02 23:02:12.070703 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:12.070619 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-g729w_1f3ab9d6-1afa-499c-80d2-a1d11981eeb2/kube-rbac-proxy-self/0.log" Nov 02 23:02:12.486520 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:12.486487 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bff846db6-xs49b_7b199a04-e491-43c9-baf3-0d9e6c663dfa/metrics-server/0.log" Nov 02 23:02:13.776244 ip-10-0-140-221 kubenswrapper[5871]: E1102 23:02:13.776196 5871 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tnqdp" containerName="extractor" Nov 02 23:02:15.102193 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:15.102150 5871 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t2vm8_425cd6ec-68c1-48ca-8243-b15082ae9553/konnectivity-agent/0.log" limit=1024 Nov 02 23:02:15.894341 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:15.894302 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gg8nx_759f8e6e-92ee-4b83-9447-949cb34a2cd1/node-exporter/0.log" Nov 02 23:02:16.323510 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:16.323434 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gg8nx_759f8e6e-92ee-4b83-9447-949cb34a2cd1/kube-rbac-proxy/0.log" Nov 02 23:02:16.741786 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:16.741750 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gg8nx_759f8e6e-92ee-4b83-9447-949cb34a2cd1/init-textfile/0.log" Nov 02 23:02:17.165446 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:17.165414 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xzrgs_ec8310e5-d852-44c1-901b-b54a6c13a798/kube-rbac-proxy-main/0.log" Nov 02 23:02:17.579694 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:17.579586 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xzrgs_ec8310e5-d852-44c1-901b-b54a6c13a798/kube-rbac-proxy-self/0.log" Nov 02 23:02:17.987791 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:17.987760 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xzrgs_ec8310e5-d852-44c1-901b-b54a6c13a798/openshift-state-metrics/0.log" Nov 02 23:02:18.411589 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:18.410508 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac460a0c-d54c-4ce4-b184-84c6c00a5db7/prometheus/0.log" Nov 02 23:02:18.820465 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:18.820387 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac460a0c-d54c-4ce4-b184-84c6c00a5db7/config-reloader/0.log" Nov 02 23:02:19.261378 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:19.261341 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac460a0c-d54c-4ce4-b184-84c6c00a5db7/thanos-sidecar/0.log" Nov 02 23:02:19.671552 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:19.671525 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac460a0c-d54c-4ce4-b184-84c6c00a5db7/kube-rbac-proxy-web/0.log" Nov 02 23:02:20.084219 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:20.084116 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac460a0c-d54c-4ce4-b184-84c6c00a5db7/kube-rbac-proxy/0.log" Nov 02 23:02:20.497692 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:20.497660 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac460a0c-d54c-4ce4-b184-84c6c00a5db7/kube-rbac-proxy-thanos/0.log" Nov 02 23:02:20.910059 ip-10-0-140-221 kubenswrapper[5871]: I1102 23:02:20.910011 5871 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ac460a0c-d54c-4ce4-b184-84c6c00a5db7/init-config-reloader/0.log"