Sep 17 16:42:12.898191 ip-10-0-132-85 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 16:42:16.616461 ip-10-0-132-85 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:42:16.616461 ip-10-0-132-85 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 16:42:16.616461 ip-10-0-132-85 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:42:16.616461 ip-10-0-132-85 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 16:42:16.616461 ip-10-0-132-85 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 16:42:16.617306 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.616553    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 16:42:16.619726 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619702    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:42:16.619726 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619721    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:42:16.619726 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619726    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:42:16.619726 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619729    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:42:16.619726 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619733    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619739    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619744    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619747    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619751    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619755    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619759    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619762    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619765    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619769    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619773    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619777    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619781    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619785    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619789    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619795    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619801    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619805    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619809    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:42:16.620049 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619813    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619817    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619822    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619826    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619830    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619846    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619852    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619856    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619860    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619865    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619869    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619874    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619878    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619883    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619887    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619892    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619897    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619902    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619906    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619927    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:42:16.620831 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619931    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619936    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619942    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619946    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619952    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619957    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619961    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619965    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619971    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619979    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619983    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619987    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619991    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.619996    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620000    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620004    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620008    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620012    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620017    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620021    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:42:16.621725 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620025    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620029    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620033    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620037    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620042    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620045    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620049    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620055    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620060    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620064    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620075    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620080    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620084    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620088    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620104    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620110    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620115    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620119    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620123    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620129    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:42:16.622602 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620133    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620137    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620141    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620146    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620150    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620155    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620159    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620163    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620167    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620172    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620176    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620181    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620185    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620189    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620193    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.620197    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621356    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621367    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621372    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621377    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:42:16.623485 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621382    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621387    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621392    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621396    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621401    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621405    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621409    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621415    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621419    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621425    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621430    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621435    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621440    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621445    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621449    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621453    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621457    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621461    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621467    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:42:16.624401 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621473    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621478    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621483    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621488    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621494    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621498    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621502    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621507    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621511    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621515    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621519    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621524    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621530    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621535    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621538    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621543    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621547    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621551    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621555    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:42:16.624988 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621560    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621564    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621570    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621574    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621578    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621582    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621587    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621591    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621595    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621599    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621603    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621607    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621611    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621615    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621620    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621624    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621628    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621633    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621637    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621642    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:42:16.625524 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621646    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621650    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621655    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621659    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621663    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621668    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621672    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621676    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621680    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621684    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621689    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621693    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621697    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621701    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621705    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621710    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621714    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621718    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621724    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621729    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:42:16.626131 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621735    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621739    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621744    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621748    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621753    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621758    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621762    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621767    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621772    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621780    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621784    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621789    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621793    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621797    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621801    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621806    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.621810    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622447    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622457    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622462    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:42:16.626646 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622466    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622472    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622477    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622482    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622487    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622491    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622495    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622501    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622506    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622510    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622514    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622518    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622523    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622527    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622532    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622536    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622540    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622545    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622549    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622553    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:42:16.627431 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622558    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622565    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622569    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622574    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622578    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622582    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622587    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622591    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622595    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622599    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622603    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622606    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622610    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622614    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622618    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622622    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622626    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622630    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622636    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622640    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:42:16.627995 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622654    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622658    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622663    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622667    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622671    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622675    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622679    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622682    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622687    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622691    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622697    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622703    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622709    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622714    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622718    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622722    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622727    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622732    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622736    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622740    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:42:16.628580 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622745    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622749    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622753    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622757    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622765    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622771    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622775    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622780    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622784    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622789    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622793    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622798    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622802    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622814    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622819    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622823    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622830    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622835    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622839    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:42:16.629089 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622843    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622848    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622853    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622858    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622862    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622866    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622871    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622876    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622880    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622884    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622888    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622892    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622897    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622901    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622905    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622930    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.622934    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623038    2565 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623099    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623108    2565 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623115    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 16:42:16.629582 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623121    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623126    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623133    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623139    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623144    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623149    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623160    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623167    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623172    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623176    2565 flags.go:64] FLAG: --cgroup-root=""
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623181    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623186    2565 flags.go:64] FLAG: --client-ca-file=""
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623191    2565 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623195    2565 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623202    2565 flags.go:64] FLAG: --cluster-domain=""
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623207    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623212    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623218    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623223    2565 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623229    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623234    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623239    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623244    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623249    2565 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 16:42:16.630172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623254    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623259    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623264    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623269    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623275    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623280    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623284    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623289    2565 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623294    2565 flags.go:64] FLAG: --enable-server="true"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623298    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623304    2565 flags.go:64] FLAG: --event-burst="100"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623309    2565 flags.go:64] FLAG: --event-qps="50"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623314    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623319    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623323    2565 flags.go:64] FLAG: --eviction-hard=""
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623333    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623338    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623343    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623348    2565 flags.go:64] FLAG: --eviction-soft=""
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623353    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623357    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623362    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623366    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623371    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623376    2565 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 16:42:16.630751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623381    2565 flags.go:64] FLAG: --feature-gates=""
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623386    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623391    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623396    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623401    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623407    2565 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623412    2565 flags.go:64] FLAG: --help="false"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623417    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-132-85.ec2.internal"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623422    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623427    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623432    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623437    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623443    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623448    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623453    2565 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623457    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623462    2565 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623467    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623472    2565 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623476    2565 flags.go:64] FLAG: --kube-reserved=""
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623481    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623485    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623492    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623497    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 16:42:16.631474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623501    2565 flags.go:64] FLAG: --lock-file=""
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623506    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623510    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623515    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623524    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623529    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623534    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623538    2565 flags.go:64] FLAG: --logging-format="text"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623545    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623550    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623555    2565 flags.go:64] FLAG: --manifest-url=""
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623559    2565 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623567    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623571    2565 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623578    2565 flags.go:64] FLAG: --max-pods="110"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623583    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623588    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623592    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623597    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623601    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623606    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623611    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623622    2565 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623627    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623632    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 16:42:16.632093 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623636    2565 flags.go:64] FLAG: --pod-cidr=""
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623641    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623649    2565 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623654    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623658    2565 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623663    2565 flags.go:64] FLAG: --port="10250"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623670    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623675    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e01ae9b1cf96128f"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623680    2565 flags.go:64] FLAG: --qos-reserved=""
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623685    2565 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623690    2565 flags.go:64] FLAG: --register-node="true"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623694    2565 flags.go:64] FLAG: --register-with-taints=""
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623700    2565 flags.go:64] FLAG: --registry-burst="10"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623705    2565 flags.go:64] FLAG: --registry-qps="5"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623709    2565 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623715    2565 flags.go:64] FLAG: --reserved-memory=""
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623720    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623725    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623730    2565 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623735    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623739    2565 flags.go:64] FLAG: --runonce="false"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623744    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623749    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623754    2565 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623758    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 16:42:16.632695 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623763    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623767    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623772    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623777    2565 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623782    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623786    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623791    2565 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623795    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623800    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623805    2565 flags.go:64] FLAG: --system-cgroups=""
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623809    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623818    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623822    2565 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623829    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623836    2565 flags.go:64] FLAG: --tls-min-version=""
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623841    2565 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623846    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623850    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623854    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623858    2565 flags.go:64] FLAG: --v="2"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623864    2565 flags.go:64] FLAG: --version="false"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623870    2565 flags.go:64] FLAG: --vmodule=""
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623878    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.623884    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624085    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:42:16.633374 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624093    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624099    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624104    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624109    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624113    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624118    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624122    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624127    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624132    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624136    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624140    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624145    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624149    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624153    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624157    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624162    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624183    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624187    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624191    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624195    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:42:16.634066 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624202    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624206    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624211    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624216    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624221    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624225    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624229    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624233    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624237    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624243    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624248    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624252    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624256    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624260    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624264    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624269    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624273    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624277    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624281    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624284    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:42:16.634557 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624289    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624294    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624299    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624303    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624307    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624311    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624317    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624323    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624328    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624334    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624339    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624343    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624349    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624354    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624358    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624362    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624368    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624373    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624377    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:42:16.635067 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624382    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624386    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624392    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624396    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624400    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624406    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624411    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624416    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624420    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624425    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624430    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624434    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624440    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624444    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624449    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624453    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624459    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624465    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624470    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:42:16.635542 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624474    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624478    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624482    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624487    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624491    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624495    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624501    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624505    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624509    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624513    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624517    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624522    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624527    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624530    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624535    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624544    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624548    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624553    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624557    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:42:16.636018 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.624561    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:42:16.636487 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.625455    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:42:16.636487 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.632623    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 16:42:16.636487 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.632640    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 16:42:16.636487 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.632660    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 16:42:16.636487 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.632668    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 16:42:16.636487 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632719    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:42:16.636487 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632725    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:42:16.636487 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632730    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:42:16.636487 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632733    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:42:16.636487 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632736    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:42:16.636487 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632739    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:42:16.636487 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632742    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:42:16.636487 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632745    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:42:16.636487 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632748    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632751    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632754    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632757    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632759    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632762    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632765    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632767    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632770    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632773    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632775    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632778    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632781    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632783    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632786    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632788    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632791    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632794    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632797    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632800    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:42:16.636851 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632804    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632808    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632811    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632814    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632816    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632819    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632822    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632824    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632827    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632830    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632832    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632835    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632838    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632840    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632843    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632846    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632848    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632851    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632853    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632857    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:42:16.637540 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632861    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632863    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632866    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632869    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632872    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632875    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632878    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632881    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632883    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632886    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632888    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632891    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632894    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632897    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632900    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632903    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632905    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632908    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632926    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:42:16.638269 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632929    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632932    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632935    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632937    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632940    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632942    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632945    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632948    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632951    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632953    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632956    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632959    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632961    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632964    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632967    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632970    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632972    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632975    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632977    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632980    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:42:16.638853 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632983    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:42:16.639478 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632985    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:42:16.639478 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632988    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:42:16.639478 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632991    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:42:16.639478 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632993    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:42:16.639478 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632996    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:42:16.639478 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.632999    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:42:16.639478 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633001    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:42:16.639478 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633005    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:42:16.639478 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633008    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:42:16.639478 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633011    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:42:16.639478 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633013    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:42:16.639478 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.633018    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:42:16.639478 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633124    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 16:42:16.639478 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633128    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633131    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633134    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633136    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633139    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633141    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633144    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633147    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633150    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633152    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633155    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633158    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633161    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633163    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633166    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633168    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633171    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633174    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633177    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633179    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 16:42:16.639827 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633182    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633184    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633187    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633189    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633192    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633195    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633197    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633202    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633205    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633207    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633210    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633213    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633215    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633218    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633220    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633223    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633227    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633231    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633234    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633236    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 16:42:16.640353 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633239    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633242    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633244    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633247    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633249    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633252    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633255    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633257    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633260    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633263    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633265    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633268    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633271    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633273    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633276    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633279    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633281    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633284    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633286    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 16:42:16.640847 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633289    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633292    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633294    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633298    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633302    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633305    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633308    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633311    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633315    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633318    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633320    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633323    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633326    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633328    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633331    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633333    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633336    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633338    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633341    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 16:42:16.641335 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633344    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633346    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633351    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633354    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633357    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633360    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633362    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633365    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633368    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633370    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633373    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633375    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633378    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633381    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633384    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633388    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633390    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633393    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633396    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 16:42:16.641803 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:16.633398    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 16:42:16.642295 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.633404    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 16:42:16.642295 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.633516    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 16:42:16.642295 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.635452    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 16:42:16.642295 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.636388    2565 server.go:1019] "Starting client certificate rotation"
Sep 17 16:42:16.642295 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.636485    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 16:42:16.642295 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.636524    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 16:42:16.642295 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.640112    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 16:42:16.642295 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.641213    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 16:42:16.645956 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.645939    2565 log.go:25] "Validated CRI v1 runtime API"
Sep 17 16:42:16.647434 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.647420    2565 log.go:25] "Validated CRI v1 image API"
Sep 17 16:42:16.648355 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.648341    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 16:42:16.651476 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.651453    2565 fs.go:135] Filesystem UUIDs: map[03d148d5-b170-4238-a1ff-536a5bf99160:/dev/nvme0n1p4 65fc710b-00c0-4f6c-8d35-e1a9ad259e65:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 17 16:42:16.651567 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.651474    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 16:42:16.656842 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.656708    2565 manager.go:216] Machine: {Timestamp:2026-09-17 16:42:16.656232452 +0000 UTC m=+2.222344165 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099463 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25dfc06ba5cf0cdf36aba3cc65e98d SystemUUID:ec25dfc0-6ba5-cf0c-df36-aba3cc65e98d BootID:3635a5f6-e149-425d-abcc-f2d3c6ebc2e8 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:de:25:ff:89 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:de:25:ff:89 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:be:85:3f:18:da Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 16:42:16.656842 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.656828    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 16:42:16.657056 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.656974    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 16:42:16.658747 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.658714    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 16:42:16.658954 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.658753    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-85.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 16:42:16.659984 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.659971    2565 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 16:42:16.660018 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.659989    2565 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 16:42:16.660018 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.659997    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 16:42:16.660083 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.660023    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 16:42:16.660116 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.660098    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 16:42:16.662774 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.662762    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 16:42:16.662890 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.662880    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 16:42:16.662957 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.662925    2565 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 16:42:16.662957 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.662934    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 16:42:16.662957 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.662947    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 16:42:16.662957 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.662955    2565 kubelet.go:398] "Adding apiserver pod source"
Sep 17 16:42:16.663120 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.662964    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 16:42:16.663540 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.663525    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 16:42:16.663618 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.663556    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 16:42:16.668306 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.667954    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 16:42:16.670240 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.670222    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 16:42:16.670314 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.670262    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 16:42:16.671567 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.671553    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 16:42:16.671607 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.671571    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 16:42:16.671607 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.671578    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 16:42:16.671607 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.671583    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 16:42:16.671607 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.671593    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 16:42:16.671607 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.671598    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 16:42:16.671740 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.671611    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 16:42:16.671740 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.671617    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 16:42:16.671740 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.671624    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 16:42:16.671740 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.671629    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 16:42:16.671740 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.671638    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 16:42:16.671740 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.671644    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 16:42:16.671740 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.671668    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 16:42:16.671740 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.671673    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 16:42:16.675293 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.675280    2565 server.go:1268] "Started kubelet"
Sep 17 16:42:16.675378 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.675352    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 16:42:16.675461 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.675374    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 16:42:16.675461 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.675443    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 16:42:16.676009 ip-10-0-132-85 systemd[1]: Started Kubernetes Kubelet.
Sep 17 16:42:16.676478 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.676004    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 16:42:16.676478 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.676310    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-85.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 16:42:16.676892 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.676874    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 16:42:16.676979 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:16.676927    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-85.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 16:42:16.676979 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:16.676929    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 16:42:16.677286 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.677275    2565 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 16:42:16.681265 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:16.681248    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 16:42:16.683755 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.683741    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 16:42:16.683828 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.683769    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 16:42:16.683874 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:16.683826    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 17 16:42:16.684333 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.684316    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 16:42:16.684333 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.684316    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 16:42:16.684454 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.684352    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 16:42:16.684511 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.684485    2565 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 16:42:16.684511 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.684496    2565 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 16:42:16.684625 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.684611    2565 factory.go:55] Registering systemd factory
Sep 17 16:42:16.684676 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.684628    2565 factory.go:223] Registration of the systemd container factory successfully
Sep 17 16:42:16.684845 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.684827    2565 factory.go:153] Registering CRI-O factory
Sep 17 16:42:16.684933 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.684849    2565 factory.go:223] Registration of the crio container factory successfully
Sep 17 16:42:16.684933 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.684896    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 16:42:16.684933 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.684932    2565 factory.go:103] Registering Raw factory
Sep 17 16:42:16.685049 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.684948    2565 manager.go:1196] Started watching for new ooms in manager
Sep 17 16:42:16.685333 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.685311    2565 manager.go:349] Starting recovery of all containers
Sep 17 16:42:16.686700 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:16.686669    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-85.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 16:42:16.686781 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:16.686750    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 16:42:16.706486 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.706466    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bm7wp"
Sep 17 16:42:16.706797 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.706778    2565 manager.go:354] Recovery completed
Sep 17 16:42:16.707478 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.707455    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bm7wp"
Sep 17 16:42:16.712507 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.712490    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:42:16.715522 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.715438    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:42:16.715522 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.715474    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:42:16.715522 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.715487    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:42:16.716326 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.716312    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 16:42:16.716326 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.716319    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:42:16.716326 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.716325    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 16:42:16.716487 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.716339    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 16:42:16.717076 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.717063    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:42:16.717127 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.717089    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:42:16.717127 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.717102    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:42:16.717588 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.717576    2565 policy_none.go:49] "None policy: Start"
Sep 17 16:42:16.717626 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.717592    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 16:42:16.717626 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.717602    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 16:42:16.718567 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.718556    2565 policy_none.go:47] "Start"
Sep 17 16:42:16.762439 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.762422    2565 manager.go:337] "Starting Device Plugin manager"
Sep 17 16:42:16.764033 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:16.762460    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 16:42:16.764033 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.762472    2565 server.go:87] "Starting device plugin registration server"
Sep 17 16:42:16.764033 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.762663    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 16:42:16.764033 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.762683    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 16:42:16.764033 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.762772    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 16:42:16.764033 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.762848    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 16:42:16.764033 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.762859    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 16:42:16.764033 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:16.763343    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 16:42:16.764033 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:16.763383    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 17 16:42:16.816894 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.816869    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 16:42:16.817995 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.817975    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 16:42:16.818075 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.818030    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 16:42:16.818075 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.818054    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 16:42:16.818171 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:16.818094    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 16:42:16.820443 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.820426    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:42:16.863171 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.863152    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:42:16.864830 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.864816    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:42:16.864895 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.864844    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:42:16.864895 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.864859    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:42:16.864895 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.864884    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-85.ec2.internal"
Sep 17 16:42:16.873779 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.873743    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-85.ec2.internal"
Sep 17 16:42:16.873779 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:16.873761    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-85.ec2.internal\": node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 17 16:42:16.888293 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:16.888272    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 17 16:42:16.918425 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.918408    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal"]
Sep 17 16:42:16.918478 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.918460    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:42:16.919214 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.919200    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:42:16.919291 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.919231    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:42:16.919291 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.919245    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:42:16.920542 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.920528    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:42:16.920673 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.920640    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 17 16:42:16.920673 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.920663    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:42:16.921223 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.921208    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:42:16.921278 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.921235    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:42:16.921278 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.921249    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:42:16.921377 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.921234    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:42:16.921408 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.921398    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:42:16.921441 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.921410    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:42:16.922311 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.922295    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal"
Sep 17 16:42:16.922393 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.922320    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 16:42:16.923004 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.922990    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 16:42:16.923071 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.923015    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 16:42:16.923071 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.923027    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientPID"
Sep 17 16:42:16.936444 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:16.936425    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-85.ec2.internal\" not found" node="ip-10-0-132-85.ec2.internal"
Sep 17 16:42:16.940509 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:16.940495    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-85.ec2.internal\" not found" node="ip-10-0-132-85.ec2.internal"
Sep 17 16:42:16.986529 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.986510    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6d096ee74184de72596674c90aed240-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal\" (UID: \"b6d096ee74184de72596674c90aed240\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 17 16:42:16.986622 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.986535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6d096ee74184de72596674c90aed240-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal\" (UID: \"b6d096ee74184de72596674c90aed240\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 17 16:42:16.986622 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:16.986552    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8bce5ac69a7a3fe95f9655d41fae9d1-config\") pod \"kube-apiserver-proxy-ip-10-0-132-85.ec2.internal\" (UID: \"b8bce5ac69a7a3fe95f9655d41fae9d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal"
Sep 17 16:42:16.988600 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:16.988578    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 17 16:42:17.087682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.087653    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6d096ee74184de72596674c90aed240-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal\" (UID: \"b6d096ee74184de72596674c90aed240\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 17 16:42:17.087797 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.087713    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6d096ee74184de72596674c90aed240-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal\" (UID: \"b6d096ee74184de72596674c90aed240\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 17 16:42:17.087797 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.087742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6d096ee74184de72596674c90aed240-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal\" (UID: \"b6d096ee74184de72596674c90aed240\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 17 16:42:17.087797 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.087753    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6d096ee74184de72596674c90aed240-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal\" (UID: \"b6d096ee74184de72596674c90aed240\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 17 16:42:17.087797 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.087779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8bce5ac69a7a3fe95f9655d41fae9d1-config\") pod \"kube-apiserver-proxy-ip-10-0-132-85.ec2.internal\" (UID: \"b8bce5ac69a7a3fe95f9655d41fae9d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal"
Sep 17 16:42:17.087955 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.087803    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8bce5ac69a7a3fe95f9655d41fae9d1-config\") pod \"kube-apiserver-proxy-ip-10-0-132-85.ec2.internal\" (UID: \"b8bce5ac69a7a3fe95f9655d41fae9d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal"
Sep 17 16:42:17.088711 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:17.088697    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 17 16:42:17.189472 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:17.189429    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 17 16:42:17.240579 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.240563    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 17 16:42:17.243707 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.243692    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal"
Sep 17 16:42:17.290253 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:17.290129    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 17 16:42:17.390600 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:17.390579    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 17 16:42:17.491008 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:17.490958    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 17 16:42:17.591469 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:17.591448    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 17 16:42:17.636619 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.636601    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 16:42:17.636956 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.636694    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 16:42:17.692423 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:17.692402    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 17 16:42:17.711988 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.711962    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 16:37:16 +0000 UTC" deadline="2028-05-11 20:17:03.967175477 +0000 UTC"
Sep 17 16:42:17.711988 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.711987    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14451h34m46.255192318s"
Sep 17 16:42:17.758784 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.758742    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:42:17.783875 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.783855    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal"
Sep 17 16:42:17.791778 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.791762    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 16:42:17.793416 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.793400    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 17 16:42:17.804646 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.804629    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 16:42:17.837490 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.837470    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:42:17.883484 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:17.883461    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d096ee74184de72596674c90aed240.slice/crio-f4b090e1a540d60def9ab9df3294d3edd8f28fa13597ac2455dc6fc9e4eaa84e WatchSource:0}: Error finding container f4b090e1a540d60def9ab9df3294d3edd8f28fa13597ac2455dc6fc9e4eaa84e: Status 404 returned error can't find the container with id f4b090e1a540d60def9ab9df3294d3edd8f28fa13597ac2455dc6fc9e4eaa84e
Sep 17 16:42:17.883849 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:17.883825    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8bce5ac69a7a3fe95f9655d41fae9d1.slice/crio-ac7489ac39335f0f0c23f8aeb15456e71700a010ef2cd39ed028226aaa8e1237 WatchSource:0}: Error finding container ac7489ac39335f0f0c23f8aeb15456e71700a010ef2cd39ed028226aaa8e1237: Status 404 returned error can't find the container with id ac7489ac39335f0f0c23f8aeb15456e71700a010ef2cd39ed028226aaa8e1237
Sep 17 16:42:17.890371 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:17.890347    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:42:18.130086 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.130067    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:42:18.664850 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.664814    2565 apiserver.go:52] "Watching apiserver"
Sep 17 16:42:18.672500 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.672479    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 16:42:18.672900 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.672874    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-6hz2q","openshift-cluster-node-tuning-operator/tuned-cqx8k","openshift-dns/node-resolver-dtmtt","openshift-image-registry/node-ca-69bv6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal","openshift-multus/network-metrics-daemon-wld7n","openshift-ovn-kubernetes/ovnkube-node-q7v29","kube-system/global-pull-secret-syncer-5llzd","kube-system/konnectivity-agent-bn474","kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss","openshift-multus/multus-additional-cni-plugins-qvxw9","openshift-multus/multus-s5lfc","openshift-network-diagnostics/network-check-target-zl97d"]
Sep 17 16:42:18.674675 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.674407    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.675953 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.675861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.676832 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.676810    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 16:42:18.677295 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.677274    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dtmtt"
Sep 17 16:42:18.677374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.677336    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 16:42:18.677566 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.677551    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 16:42:18.678276 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.678255    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 16:42:18.678533 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.678517    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 16:42:18.678723 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.678702    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 16:42:18.678723 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.678718    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 16:42:18.678837 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.678520    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 16:42:18.679054 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.679037    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nxbq4\""
Sep 17 16:42:18.679259 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.679245    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vnmqk\""
Sep 17 16:42:18.679465 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.679449    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-69bv6"
Sep 17 16:42:18.680138 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.679716    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:18.680138 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:18.679805    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:18.680138 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.680012    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 16:42:18.680138 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.680051    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 16:42:18.680364 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.680202    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 16:42:18.680364 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.680202    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pdkjt\""
Sep 17 16:42:18.680879 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.680761    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:18.680879 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:18.680822    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:18.681707 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.681687    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 16:42:18.681941 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.681894    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ch5ht\""
Sep 17 16:42:18.682165 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.681894    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 16:42:18.682360 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.682342    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 16:42:18.682433 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.682358    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bn474"
Sep 17 16:42:18.684110 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.683885    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.685973 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.684775    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 16:42:18.685973 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.684878    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4hdg8\""
Sep 17 16:42:18.685973 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.684929    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 16:42:18.685973 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.684973    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 16:42:18.685973 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.685631    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.686265 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.686226    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 16:42:18.686374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.686349    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 16:42:18.686458 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.686438    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 16:42:18.686545 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.686525    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-68bf2\""
Sep 17 16:42:18.686653 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.686582    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 16:42:18.687392 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.687356    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:18.688961 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:18.687647    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:18.690509 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.690455    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8lmmr\""
Sep 17 16:42:18.690607 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.690523    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 16:42:18.690791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.690776    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:42:18.691246 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.690996    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6hz2q"
Sep 17 16:42:18.692272 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.692251    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.693997 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.693797    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 16:42:18.694128 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.694107    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 16:42:18.694196 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.694181    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 16:42:18.694590 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.694561    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v9nt8\""
Sep 17 16:42:18.695118 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.694856    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xnzzt\""
Sep 17 16:42:18.695118 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.694859    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 16:42:18.695118 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.694858    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 16:42:18.696929 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.696898    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-run-openvswitch\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.697020 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.696942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-multus-cni-dir\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.697020 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.696964    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-sys\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.697020 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.696986    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqqbr\" (UniqueName: \"kubernetes.io/projected/ffd47162-fa3a-4c51-a370-3ffe004cf33a-kube-api-access-lqqbr\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.697020 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-cni-bin\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.697217 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697031    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-socket-dir\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.697217 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697053    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-slash\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.697217 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697077    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/700d8b01-1d3b-4b49-bb6a-696687e029bb-ovnkube-config\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.697217 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/700d8b01-1d3b-4b49-bb6a-696687e029bb-ovn-node-metrics-cert\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.697217 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697130    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-527zn\" (UniqueName: \"kubernetes.io/projected/4a45b30c-1f89-4643-84b2-189a36d6e0f8-kube-api-access-527zn\") pod \"node-resolver-dtmtt\" (UID: \"4a45b30c-1f89-4643-84b2-189a36d6e0f8\") " pod="openshift-dns/node-resolver-dtmtt"
Sep 17 16:42:18.697217 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697154    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-hostroot\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.697217 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697166    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 16:42:18.697217 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697177    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-sysconfig\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.697217 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj2m6\" (UniqueName: \"kubernetes.io/projected/307aa194-969e-445a-8f44-22359d43a8b5-kube-api-access-dj2m6\") pod \"node-ca-69bv6\" (UID: \"307aa194-969e-445a-8f44-22359d43a8b5\") " pod="openshift-image-registry/node-ca-69bv6"
Sep 17 16:42:18.697595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697233    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-run-netns\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.697595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697255    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-lib-modules\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.697595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697295    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/307aa194-969e-445a-8f44-22359d43a8b5-host\") pod \"node-ca-69bv6\" (UID: \"307aa194-969e-445a-8f44-22359d43a8b5\") " pod="openshift-image-registry/node-ca-69bv6"
Sep 17 16:42:18.697595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4tqx\" (UniqueName: \"kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx\") pod \"network-check-target-zl97d\" (UID: \"c0a61d36-0e50-4539-ac3f-d5b6c9c3e287\") " pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:18.697595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697353    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/700d8b01-1d3b-4b49-bb6a-696687e029bb-ovnkube-script-lib\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.697595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697378    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-systemd-units\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.697595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697399    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-var-lib-openvswitch\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.697595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-run-ovn\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.697595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.697595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697538    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.697595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697563    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a45b30c-1f89-4643-84b2-189a36d6e0f8-tmp-dir\") pod \"node-resolver-dtmtt\" (UID: \"4a45b30c-1f89-4643-84b2-189a36d6e0f8\") " pod="openshift-dns/node-resolver-dtmtt"
Sep 17 16:42:18.697595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697586    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-cnibin\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697608    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-tuned\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffd47162-fa3a-4c51-a370-3ffe004cf33a-tmp\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697674    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-device-dir\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ce775b0-037d-4846-a3b6-f8bf2f76f293-agent-certs\") pod \"konnectivity-agent-bn474\" (UID: \"3ce775b0-037d-4846-a3b6-f8bf2f76f293\") " pod="kube-system/konnectivity-agent-bn474"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697753    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-var-lib-cni-bin\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697782    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-var-lib-kubelet\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697811    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-etc-kubernetes\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697838    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-var-lib-kubelet\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697859    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-node-log\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697881    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-kubernetes\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697902    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-etc-openvswitch\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.697939    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-cni-netd\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698003    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a45b30c-1f89-4643-84b2-189a36d6e0f8-hosts-file\") pod \"node-resolver-dtmtt\" (UID: \"4a45b30c-1f89-4643-84b2-189a36d6e0f8\") " pod="openshift-dns/node-resolver-dtmtt"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-multus-conf-dir\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-kubelet\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.698082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698082    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-sys-fs\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698106    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-multus-socket-dir-parent\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698130    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57b77\" (UniqueName: \"kubernetes.io/projected/aef9115b-1284-4e79-a8e3-9e8d1778a76e-kube-api-access-57b77\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698153    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-systemd\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698175    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-run\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698198    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698220    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dbgf\" (UniqueName: \"kubernetes.io/projected/716feac9-5ed2-4b28-9ede-a210a4871e3d-kube-api-access-6dbgf\") pod \"network-metrics-daemon-wld7n\" (UID: \"716feac9-5ed2-4b28-9ede-a210a4871e3d\") " pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698260    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-modprobe-d\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698292    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-run-netns\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698316    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp5bj\" (UniqueName: \"kubernetes.io/projected/700d8b01-1d3b-4b49-bb6a-696687e029bb-kube-api-access-fp5bj\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698340    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aef9115b-1284-4e79-a8e3-9e8d1778a76e-cni-binary-copy\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698365    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-run-multus-certs\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698387    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/307aa194-969e-445a-8f44-22359d43a8b5-serviceca\") pod \"node-ca-69bv6\" (UID: \"307aa194-969e-445a-8f44-22359d43a8b5\") " pod="openshift-image-registry/node-ca-69bv6"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698427    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs\") pod \"network-metrics-daemon-wld7n\" (UID: \"716feac9-5ed2-4b28-9ede-a210a4871e3d\") " pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698461    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-sysctl-conf\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698482    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-host\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-run-systemd\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.698808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698529    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-log-socket\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.699664 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698552    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-registration-dir\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.699664 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698575    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svl2r\" (UniqueName: \"kubernetes.io/projected/72016799-7f64-4649-b19d-28618010a6a3-kube-api-access-svl2r\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.699664 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698608    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-sysctl-d\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.699664 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698673    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ce775b0-037d-4846-a3b6-f8bf2f76f293-konnectivity-ca\") pod \"konnectivity-agent-bn474\" (UID: \"3ce775b0-037d-4846-a3b6-f8bf2f76f293\") " pod="kube-system/konnectivity-agent-bn474"
Sep 17 16:42:18.699664 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698704    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-run-k8s-cni-cncf-io\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.699664 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698733    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-var-lib-cni-multus\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.699664 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698762    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aef9115b-1284-4e79-a8e3-9e8d1778a76e-multus-daemon-config\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.699664 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698793    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/700d8b01-1d3b-4b49-bb6a-696687e029bb-env-overrides\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.699664 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.698830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-system-cni-dir\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.722878 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.722854    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r9nrk"
Sep 17 16:42:18.728406 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.728385    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r9nrk"
Sep 17 16:42:18.785636 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.785613    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 16:42:18.799544 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-systemd-units\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.799648 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799558    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-var-lib-openvswitch\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.799648 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799589    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-run-ovn\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.799648 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799615    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.799648 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799639    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.799786 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a45b30c-1f89-4643-84b2-189a36d6e0f8-tmp-dir\") pod \"node-resolver-dtmtt\" (UID: \"4a45b30c-1f89-4643-84b2-189a36d6e0f8\") " pod="openshift-dns/node-resolver-dtmtt"
Sep 17 16:42:18.799786 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799692    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-cnibin\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.799786 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-tuned\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.799786 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799737    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffd47162-fa3a-4c51-a370-3ffe004cf33a-tmp\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.799786 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.799952 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-device-dir\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.799952 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ce775b0-037d-4846-a3b6-f8bf2f76f293-agent-certs\") pod \"konnectivity-agent-bn474\" (UID: \"3ce775b0-037d-4846-a3b6-f8bf2f76f293\") " pod="kube-system/konnectivity-agent-bn474"
Sep 17 16:42:18.799952 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799832    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-var-lib-cni-bin\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.799952 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799856    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-var-lib-kubelet\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.799952 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799878    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-etc-kubernetes\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.799952 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799899    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-var-lib-kubelet\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.799952 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799936    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-node-log\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.800175 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799964    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a6a6976-08a4-4f97-9abe-a558269301a0-host-slash\") pod \"iptables-alerter-6hz2q\" (UID: \"8a6a6976-08a4-4f97-9abe-a558269301a0\") " pod="openshift-network-operator/iptables-alerter-6hz2q"
Sep 17 16:42:18.800175 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.799990    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7m4d\" (UniqueName: \"kubernetes.io/projected/8a6a6976-08a4-4f97-9abe-a558269301a0-kube-api-access-f7m4d\") pod \"iptables-alerter-6hz2q\" (UID: \"8a6a6976-08a4-4f97-9abe-a558269301a0\") " pod="openshift-network-operator/iptables-alerter-6hz2q"
Sep 17 16:42:18.800175 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4b8badb-da9b-4399-96ad-aa05263c9542-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.800175 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800048    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-kubernetes\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.800175 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800081    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-etc-openvswitch\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.800175 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-cni-netd\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.800175 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800133    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a45b30c-1f89-4643-84b2-189a36d6e0f8-hosts-file\") pod \"node-resolver-dtmtt\" (UID: \"4a45b30c-1f89-4643-84b2-189a36d6e0f8\") " pod="openshift-dns/node-resolver-dtmtt"
Sep 17 16:42:18.800175 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800156    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-multus-conf-dir\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.800417 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-kubelet\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.800417 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-sys-fs\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.800417 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-multus-socket-dir-parent\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.800417 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800246    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57b77\" (UniqueName: \"kubernetes.io/projected/aef9115b-1284-4e79-a8e3-9e8d1778a76e-kube-api-access-57b77\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.800417 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-systemd\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.800417 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800289    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-run\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.800417 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.800417 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dbgf\" (UniqueName: \"kubernetes.io/projected/716feac9-5ed2-4b28-9ede-a210a4871e3d-kube-api-access-6dbgf\") pod \"network-metrics-daemon-wld7n\" (UID: \"716feac9-5ed2-4b28-9ede-a210a4871e3d\") " pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:18.800417 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800368    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4b8badb-da9b-4399-96ad-aa05263c9542-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.800417 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800397    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-modprobe-d\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.800737 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-run-netns\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.800737 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800446    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp5bj\" (UniqueName: \"kubernetes.io/projected/700d8b01-1d3b-4b49-bb6a-696687e029bb-kube-api-access-fp5bj\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.800737 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aef9115b-1284-4e79-a8e3-9e8d1778a76e-cni-binary-copy\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.800737 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800498    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-run-multus-certs\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.800737 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800523    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmdkd\" (UniqueName: \"kubernetes.io/projected/c4b8badb-da9b-4399-96ad-aa05263c9542-kube-api-access-zmdkd\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.800737 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800546    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/307aa194-969e-445a-8f44-22359d43a8b5-serviceca\") pod \"node-ca-69bv6\" (UID: \"307aa194-969e-445a-8f44-22359d43a8b5\") " pod="openshift-image-registry/node-ca-69bv6"
Sep 17 16:42:18.800737 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs\") pod \"network-metrics-daemon-wld7n\" (UID: \"716feac9-5ed2-4b28-9ede-a210a4871e3d\") " pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:18.800737 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800605    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-sysctl-conf\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.800737 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-host\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.800737 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800660    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-run-systemd\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.800737 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-log-socket\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.800737 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-registration-dir\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.800737 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800733    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svl2r\" (UniqueName: \"kubernetes.io/projected/72016799-7f64-4649-b19d-28618010a6a3-kube-api-access-svl2r\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-sysctl-d\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800780    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ce775b0-037d-4846-a3b6-f8bf2f76f293-konnectivity-ca\") pod \"konnectivity-agent-bn474\" (UID: \"3ce775b0-037d-4846-a3b6-f8bf2f76f293\") " pod="kube-system/konnectivity-agent-bn474"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800803    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-run-k8s-cni-cncf-io\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800828    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-var-lib-cni-multus\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aef9115b-1284-4e79-a8e3-9e8d1778a76e-multus-daemon-config\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800885    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a6a6976-08a4-4f97-9abe-a558269301a0-iptables-alerter-script\") pod \"iptables-alerter-6hz2q\" (UID: \"8a6a6976-08a4-4f97-9abe-a558269301a0\") " pod="openshift-network-operator/iptables-alerter-6hz2q"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800927    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/700d8b01-1d3b-4b49-bb6a-696687e029bb-env-overrides\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-system-cni-dir\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.800976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-run-openvswitch\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-multus-cni-dir\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-sys\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqqbr\" (UniqueName: \"kubernetes.io/projected/ffd47162-fa3a-4c51-a370-3ffe004cf33a-kube-api-access-lqqbr\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801071    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-cni-bin\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-socket-dir\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801118    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/70f3faf3-b150-468f-abb7-40f2d221164e-dbus\") pod \"global-pull-secret-syncer-5llzd\" (UID: \"70f3faf3-b150-468f-abb7-40f2d221164e\") " pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801142    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4b8badb-da9b-4399-96ad-aa05263c9542-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.801290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-slash\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.802167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801193    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/700d8b01-1d3b-4b49-bb6a-696687e029bb-ovnkube-config\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.802167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801218    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/700d8b01-1d3b-4b49-bb6a-696687e029bb-ovn-node-metrics-cert\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.802167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801243    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-527zn\" (UniqueName: \"kubernetes.io/projected/4a45b30c-1f89-4643-84b2-189a36d6e0f8-kube-api-access-527zn\") pod \"node-resolver-dtmtt\" (UID: \"4a45b30c-1f89-4643-84b2-189a36d6e0f8\") " pod="openshift-dns/node-resolver-dtmtt"
Sep 17 16:42:18.802167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-hostroot\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.802167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801381    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-var-lib-openvswitch\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.802167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-modprobe-d\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.802167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801498    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/70f3faf3-b150-468f-abb7-40f2d221164e-kubelet-config\") pod \"global-pull-secret-syncer-5llzd\" (UID: \"70f3faf3-b150-468f-abb7-40f2d221164e\") " pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:18.802167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801504    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-node-log\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.802167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801544    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-etc-kubernetes\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.802167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801575    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-kubernetes\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.802167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-hostroot\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.802167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801604    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-etc-openvswitch\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.802167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801653    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-sysctl-d\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.802167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801821    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 16:42:18.802167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.801901    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-var-lib-kubelet\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.802167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802066    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ce775b0-037d-4846-a3b6-f8bf2f76f293-konnectivity-ca\") pod \"konnectivity-agent-bn474\" (UID: \"3ce775b0-037d-4846-a3b6-f8bf2f76f293\") " pod="kube-system/konnectivity-agent-bn474"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802235    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-run\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802275    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-cni-netd\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802301    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/307aa194-969e-445a-8f44-22359d43a8b5-serviceca\") pod \"node-ca-69bv6\" (UID: \"307aa194-969e-445a-8f44-22359d43a8b5\") " pod="openshift-image-registry/node-ca-69bv6"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aef9115b-1284-4e79-a8e3-9e8d1778a76e-cni-binary-copy\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802317    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a45b30c-1f89-4643-84b2-189a36d6e0f8-hosts-file\") pod \"node-resolver-dtmtt\" (UID: \"4a45b30c-1f89-4643-84b2-189a36d6e0f8\") " pod="openshift-dns/node-resolver-dtmtt"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802350    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-kubelet\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802351    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-run-multus-certs\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-multus-conf-dir\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802402    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-sys-fs\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802446    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-multus-socket-dir-parent\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802638    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/700d8b01-1d3b-4b49-bb6a-696687e029bb-ovnkube-config\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802694    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-run-k8s-cni-cncf-io\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802730    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-run-ovn\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802733    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/700d8b01-1d3b-4b49-bb6a-696687e029bb-env-overrides\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802697    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802769    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-systemd\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.803000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802859    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.803791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a45b30c-1f89-4643-84b2-189a36d6e0f8-tmp-dir\") pod \"node-resolver-dtmtt\" (UID: \"4a45b30c-1f89-4643-84b2-189a36d6e0f8\") " pod="openshift-dns/node-resolver-dtmtt"
Sep 17 16:42:18.803791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803195    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aef9115b-1284-4e79-a8e3-9e8d1778a76e-multus-daemon-config\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.803791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803239    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-slash\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.803791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803272    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-var-lib-kubelet\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.803791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.803791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803333    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.803791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803355    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-cnibin\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.803791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803397    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-log-socket\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.803791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803438    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-system-cni-dir\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.803791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-run-openvswitch\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.803791 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:18.803531    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:18.803791 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:18.803627    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs podName:716feac9-5ed2-4b28-9ede-a210a4871e3d nodeName:}" failed. No retries permitted until 2026-09-17 16:42:19.303596814 +0000 UTC m=+4.869708515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs") pod "network-metrics-daemon-wld7n" (UID: "716feac9-5ed2-4b28-9ede-a210a4871e3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:18.803791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803671    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-var-lib-cni-bin\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.803791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803721    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-registration-dir\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.803791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803756    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-cni-bin\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803842    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-socket-dir\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803882    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-multus-cni-dir\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-sysconfig\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.802736    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-var-lib-cni-multus\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.803951    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-sysconfig\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804013    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj2m6\" (UniqueName: \"kubernetes.io/projected/307aa194-969e-445a-8f44-22359d43a8b5-kube-api-access-dj2m6\") pod \"node-ca-69bv6\" (UID: \"307aa194-969e-445a-8f44-22359d43a8b5\") " pod="openshift-image-registry/node-ca-69bv6"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804041    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-run-netns\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-lib-modules\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/307aa194-969e-445a-8f44-22359d43a8b5-host\") pod \"node-ca-69bv6\" (UID: \"307aa194-969e-445a-8f44-22359d43a8b5\") " pod="openshift-image-registry/node-ca-69bv6"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804125    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b8badb-da9b-4399-96ad-aa05263c9542-system-cni-dir\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804153    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4b8badb-da9b-4399-96ad-aa05263c9542-cnibin\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804180    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4b8badb-da9b-4399-96ad-aa05263c9542-cni-binary-copy\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804206    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tqx\" (UniqueName: \"kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx\") pod \"network-check-target-zl97d\" (UID: \"c0a61d36-0e50-4539-ac3f-d5b6c9c3e287\") " pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/700d8b01-1d3b-4b49-bb6a-696687e029bb-ovnkube-script-lib\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804282    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-sysctl-conf\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret\") pod \"global-pull-secret-syncer-5llzd\" (UID: \"70f3faf3-b150-468f-abb7-40f2d221164e\") " pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:18.804534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804327    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-sys\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.805339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804362    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-host\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.805339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804401    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/72016799-7f64-4649-b19d-28618010a6a3-device-dir\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.805339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804557    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-run-systemd\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.805339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804645    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/307aa194-969e-445a-8f44-22359d43a8b5-host\") pod \"node-ca-69bv6\" (UID: \"307aa194-969e-445a-8f44-22359d43a8b5\") " pod="openshift-image-registry/node-ca-69bv6"
Sep 17 16:42:18.805339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804680    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-host-run-netns\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.805339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804713    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/700d8b01-1d3b-4b49-bb6a-696687e029bb-systemd-units\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.805339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804746    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aef9115b-1284-4e79-a8e3-9e8d1778a76e-host-run-netns\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.805339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804825    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffd47162-fa3a-4c51-a370-3ffe004cf33a-lib-modules\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.805339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.804826    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/700d8b01-1d3b-4b49-bb6a-696687e029bb-ovnkube-script-lib\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.807997 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.807603    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffd47162-fa3a-4c51-a370-3ffe004cf33a-tmp\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.807997 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.807715    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/700d8b01-1d3b-4b49-bb6a-696687e029bb-ovn-node-metrics-cert\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.807997 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.807889    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ce775b0-037d-4846-a3b6-f8bf2f76f293-agent-certs\") pod \"konnectivity-agent-bn474\" (UID: \"3ce775b0-037d-4846-a3b6-f8bf2f76f293\") " pod="kube-system/konnectivity-agent-bn474"
Sep 17 16:42:18.808299 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.808280    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ffd47162-fa3a-4c51-a370-3ffe004cf33a-etc-tuned\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.810680 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:18.810646    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:18.810680 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:18.810673    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:18.810833 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:18.810686    2565 projected.go:196] Error preparing data for projected volume kube-api-access-p4tqx for pod openshift-network-diagnostics/network-check-target-zl97d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:18.810833 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:18.810752    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx podName:c0a61d36-0e50-4539-ac3f-d5b6c9c3e287 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:19.310735674 +0000 UTC m=+4.876847393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p4tqx" (UniqueName: "kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx") pod "network-check-target-zl97d" (UID: "c0a61d36-0e50-4539-ac3f-d5b6c9c3e287") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:18.812367 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.812260    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-527zn\" (UniqueName: \"kubernetes.io/projected/4a45b30c-1f89-4643-84b2-189a36d6e0f8-kube-api-access-527zn\") pod \"node-resolver-dtmtt\" (UID: \"4a45b30c-1f89-4643-84b2-189a36d6e0f8\") " pod="openshift-dns/node-resolver-dtmtt"
Sep 17 16:42:18.813058 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.812908    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp5bj\" (UniqueName: \"kubernetes.io/projected/700d8b01-1d3b-4b49-bb6a-696687e029bb-kube-api-access-fp5bj\") pod \"ovnkube-node-q7v29\" (UID: \"700d8b01-1d3b-4b49-bb6a-696687e029bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.815334 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.815308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57b77\" (UniqueName: \"kubernetes.io/projected/aef9115b-1284-4e79-a8e3-9e8d1778a76e-kube-api-access-57b77\") pod \"multus-s5lfc\" (UID: \"aef9115b-1284-4e79-a8e3-9e8d1778a76e\") " pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:18.817075 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.817055    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dbgf\" (UniqueName: \"kubernetes.io/projected/716feac9-5ed2-4b28-9ede-a210a4871e3d-kube-api-access-6dbgf\") pod \"network-metrics-daemon-wld7n\" (UID: \"716feac9-5ed2-4b28-9ede-a210a4871e3d\") " pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:18.817198 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.817172    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj2m6\" (UniqueName: \"kubernetes.io/projected/307aa194-969e-445a-8f44-22359d43a8b5-kube-api-access-dj2m6\") pod \"node-ca-69bv6\" (UID: \"307aa194-969e-445a-8f44-22359d43a8b5\") " pod="openshift-image-registry/node-ca-69bv6"
Sep 17 16:42:18.817569 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.817545    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svl2r\" (UniqueName: \"kubernetes.io/projected/72016799-7f64-4649-b19d-28618010a6a3-kube-api-access-svl2r\") pod \"aws-ebs-csi-driver-node-pfqss\" (UID: \"72016799-7f64-4649-b19d-28618010a6a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:18.822949 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.821561    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqqbr\" (UniqueName: \"kubernetes.io/projected/ffd47162-fa3a-4c51-a370-3ffe004cf33a-kube-api-access-lqqbr\") pod \"tuned-cqx8k\" (UID: \"ffd47162-fa3a-4c51-a370-3ffe004cf33a\") " pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:18.827139 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.827049    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal" event={"ID":"b8bce5ac69a7a3fe95f9655d41fae9d1","Type":"ContainerStarted","Data":"ac7489ac39335f0f0c23f8aeb15456e71700a010ef2cd39ed028226aaa8e1237"}
Sep 17 16:42:18.829392 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.829373    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal" event={"ID":"b6d096ee74184de72596674c90aed240","Type":"ContainerStarted","Data":"f4b090e1a540d60def9ab9df3294d3edd8f28fa13597ac2455dc6fc9e4eaa84e"}
Sep 17 16:42:18.849759 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.849739    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 16:42:18.905222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.904846    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7m4d\" (UniqueName: \"kubernetes.io/projected/8a6a6976-08a4-4f97-9abe-a558269301a0-kube-api-access-f7m4d\") pod \"iptables-alerter-6hz2q\" (UID: \"8a6a6976-08a4-4f97-9abe-a558269301a0\") " pod="openshift-network-operator/iptables-alerter-6hz2q"
Sep 17 16:42:18.905222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.904885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4b8badb-da9b-4399-96ad-aa05263c9542-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.905222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.904933    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4b8badb-da9b-4399-96ad-aa05263c9542-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.905222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.904963    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmdkd\" (UniqueName: \"kubernetes.io/projected/c4b8badb-da9b-4399-96ad-aa05263c9542-kube-api-access-zmdkd\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.905222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a6a6976-08a4-4f97-9abe-a558269301a0-iptables-alerter-script\") pod \"iptables-alerter-6hz2q\" (UID: \"8a6a6976-08a4-4f97-9abe-a558269301a0\") " pod="openshift-network-operator/iptables-alerter-6hz2q"
Sep 17 16:42:18.905222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/70f3faf3-b150-468f-abb7-40f2d221164e-dbus\") pod \"global-pull-secret-syncer-5llzd\" (UID: \"70f3faf3-b150-468f-abb7-40f2d221164e\") " pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:18.905222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905060    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4b8badb-da9b-4399-96ad-aa05263c9542-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.905222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4b8badb-da9b-4399-96ad-aa05263c9542-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.905222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/70f3faf3-b150-468f-abb7-40f2d221164e-kubelet-config\") pod \"global-pull-secret-syncer-5llzd\" (UID: \"70f3faf3-b150-468f-abb7-40f2d221164e\") " pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:18.905222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905161    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b8badb-da9b-4399-96ad-aa05263c9542-system-cni-dir\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.905222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905184    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4b8badb-da9b-4399-96ad-aa05263c9542-cnibin\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.905222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905201    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/70f3faf3-b150-468f-abb7-40f2d221164e-kubelet-config\") pod \"global-pull-secret-syncer-5llzd\" (UID: \"70f3faf3-b150-468f-abb7-40f2d221164e\") " pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:18.905222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4b8badb-da9b-4399-96ad-aa05263c9542-cni-binary-copy\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.905222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905157    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/70f3faf3-b150-468f-abb7-40f2d221164e-dbus\") pod \"global-pull-secret-syncer-5llzd\" (UID: \"70f3faf3-b150-468f-abb7-40f2d221164e\") " pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:18.905941 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905244    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret\") pod \"global-pull-secret-syncer-5llzd\" (UID: \"70f3faf3-b150-468f-abb7-40f2d221164e\") " pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:18.905941 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905260    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b8badb-da9b-4399-96ad-aa05263c9542-system-cni-dir\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.905941 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a6a6976-08a4-4f97-9abe-a558269301a0-host-slash\") pod \"iptables-alerter-6hz2q\" (UID: \"8a6a6976-08a4-4f97-9abe-a558269301a0\") " pod="openshift-network-operator/iptables-alerter-6hz2q"
Sep 17 16:42:18.905941 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905286    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4b8badb-da9b-4399-96ad-aa05263c9542-cnibin\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.905941 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905339    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a6a6976-08a4-4f97-9abe-a558269301a0-host-slash\") pod \"iptables-alerter-6hz2q\" (UID: \"8a6a6976-08a4-4f97-9abe-a558269301a0\") " pod="openshift-network-operator/iptables-alerter-6hz2q"
Sep 17 16:42:18.905941 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:18.905372    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:18.905941 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:18.905422    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret podName:70f3faf3-b150-468f-abb7-40f2d221164e nodeName:}" failed. No retries permitted until 2026-09-17 16:42:19.40540679 +0000 UTC m=+4.971518507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret") pod "global-pull-secret-syncer-5llzd" (UID: "70f3faf3-b150-468f-abb7-40f2d221164e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:18.905941 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4b8badb-da9b-4399-96ad-aa05263c9542-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.905941 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905565    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4b8badb-da9b-4399-96ad-aa05263c9542-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.905941 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905635    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a6a6976-08a4-4f97-9abe-a558269301a0-iptables-alerter-script\") pod \"iptables-alerter-6hz2q\" (UID: \"8a6a6976-08a4-4f97-9abe-a558269301a0\") " pod="openshift-network-operator/iptables-alerter-6hz2q"
Sep 17 16:42:18.905941 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.905703    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4b8badb-da9b-4399-96ad-aa05263c9542-cni-binary-copy\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.916354 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.916304    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7m4d\" (UniqueName: \"kubernetes.io/projected/8a6a6976-08a4-4f97-9abe-a558269301a0-kube-api-access-f7m4d\") pod \"iptables-alerter-6hz2q\" (UID: \"8a6a6976-08a4-4f97-9abe-a558269301a0\") " pod="openshift-network-operator/iptables-alerter-6hz2q"
Sep 17 16:42:18.917001 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.916978    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmdkd\" (UniqueName: \"kubernetes.io/projected/c4b8badb-da9b-4399-96ad-aa05263c9542-kube-api-access-zmdkd\") pod \"multus-additional-cni-plugins-qvxw9\" (UID: \"c4b8badb-da9b-4399-96ad-aa05263c9542\") " pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:18.991053 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.991028    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:18.998522 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:18.998380    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss"
Sep 17 16:42:19.000758 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:19.000651    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700d8b01_1d3b_4b49_bb6a_696687e029bb.slice/crio-72b45b134a7922d0eb688c924e4d57cf15896eea5d3a8b3766eef1d360986390 WatchSource:0}: Error finding container 72b45b134a7922d0eb688c924e4d57cf15896eea5d3a8b3766eef1d360986390: Status 404 returned error can't find the container with id 72b45b134a7922d0eb688c924e4d57cf15896eea5d3a8b3766eef1d360986390
Sep 17 16:42:19.007040 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.006952    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dtmtt"
Sep 17 16:42:19.007789 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:19.007765    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72016799_7f64_4649_b19d_28618010a6a3.slice/crio-c83ec1df71ea7c723a88cae247c7ae03c1e1d0e969ab1631858158be1d931822 WatchSource:0}: Error finding container c83ec1df71ea7c723a88cae247c7ae03c1e1d0e969ab1631858158be1d931822: Status 404 returned error can't find the container with id c83ec1df71ea7c723a88cae247c7ae03c1e1d0e969ab1631858158be1d931822
Sep 17 16:42:19.014156 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.014116    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-69bv6"
Sep 17 16:42:19.017704 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:19.017667    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a45b30c_1f89_4643_84b2_189a36d6e0f8.slice/crio-272bb546f66e1bea0e07323e0e6abc9243877e4300f307db171a663220aaeeda WatchSource:0}: Error finding container 272bb546f66e1bea0e07323e0e6abc9243877e4300f307db171a663220aaeeda: Status 404 returned error can't find the container with id 272bb546f66e1bea0e07323e0e6abc9243877e4300f307db171a663220aaeeda
Sep 17 16:42:19.021727 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.021551    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bn474"
Sep 17 16:42:19.025022 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:19.024991    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307aa194_969e_445a_8f44_22359d43a8b5.slice/crio-cdd82e1b770468492f855bcb7a5aabf39f226b80f630ec6578c97616d658fac0 WatchSource:0}: Error finding container cdd82e1b770468492f855bcb7a5aabf39f226b80f630ec6578c97616d658fac0: Status 404 returned error can't find the container with id cdd82e1b770468492f855bcb7a5aabf39f226b80f630ec6578c97616d658fac0
Sep 17 16:42:19.030297 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:19.030274    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce775b0_037d_4846_a3b6_f8bf2f76f293.slice/crio-9b71eaa2afd0b90b4a6bc72a8487c83c50bf3ef59dfc24d3e967c1ffba583346 WatchSource:0}: Error finding container 9b71eaa2afd0b90b4a6bc72a8487c83c50bf3ef59dfc24d3e967c1ffba583346: Status 404 returned error can't find the container with id 9b71eaa2afd0b90b4a6bc72a8487c83c50bf3ef59dfc24d3e967c1ffba583346
Sep 17 16:42:19.037981 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.037945    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s5lfc"
Sep 17 16:42:19.045703 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.045680    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cqx8k"
Sep 17 16:42:19.046775 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:19.046752    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef9115b_1284_4e79_a8e3_9e8d1778a76e.slice/crio-006ba8e02eb82d7d1aa801f853c3821c9754e9ce7713204a0f7fb370aae39e5c WatchSource:0}: Error finding container 006ba8e02eb82d7d1aa801f853c3821c9754e9ce7713204a0f7fb370aae39e5c: Status 404 returned error can't find the container with id 006ba8e02eb82d7d1aa801f853c3821c9754e9ce7713204a0f7fb370aae39e5c
Sep 17 16:42:19.054966 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.054949    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6hz2q"
Sep 17 16:42:19.055091 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:19.055062    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd47162_fa3a_4c51_a370_3ffe004cf33a.slice/crio-d20f3acb23af6e9eff08e0b6b312cd81d02227baf75a3926c9ec00f5ca8e29a0 WatchSource:0}: Error finding container d20f3acb23af6e9eff08e0b6b312cd81d02227baf75a3926c9ec00f5ca8e29a0: Status 404 returned error can't find the container with id d20f3acb23af6e9eff08e0b6b312cd81d02227baf75a3926c9ec00f5ca8e29a0
Sep 17 16:42:19.062098 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.062080    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qvxw9"
Sep 17 16:42:19.064454 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:19.064423    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6a6976_08a4_4f97_9abe_a558269301a0.slice/crio-626f5ca369ae2c33c82c59e853b53fd3b5957092476a37b85cdbfb00cea21942 WatchSource:0}: Error finding container 626f5ca369ae2c33c82c59e853b53fd3b5957092476a37b85cdbfb00cea21942: Status 404 returned error can't find the container with id 626f5ca369ae2c33c82c59e853b53fd3b5957092476a37b85cdbfb00cea21942
Sep 17 16:42:19.071824 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:42:19.071803    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b8badb_da9b_4399_96ad_aa05263c9542.slice/crio-931a31f689066c9ee62414069e9e0641b7fa3b2d116cf523ba58cc20a85042b6 WatchSource:0}: Error finding container 931a31f689066c9ee62414069e9e0641b7fa3b2d116cf523ba58cc20a85042b6: Status 404 returned error can't find the container with id 931a31f689066c9ee62414069e9e0641b7fa3b2d116cf523ba58cc20a85042b6
Sep 17 16:42:19.308393 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.308311    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs\") pod \"network-metrics-daemon-wld7n\" (UID: \"716feac9-5ed2-4b28-9ede-a210a4871e3d\") " pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:19.308539 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:19.308467    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:19.308597 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:19.308539    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs podName:716feac9-5ed2-4b28-9ede-a210a4871e3d nodeName:}" failed. No retries permitted until 2026-09-17 16:42:20.308519525 +0000 UTC m=+5.874631245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs") pod "network-metrics-daemon-wld7n" (UID: "716feac9-5ed2-4b28-9ede-a210a4871e3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:19.409634 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.409606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tqx\" (UniqueName: \"kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx\") pod \"network-check-target-zl97d\" (UID: \"c0a61d36-0e50-4539-ac3f-d5b6c9c3e287\") " pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:19.409634 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.409640    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret\") pod \"global-pull-secret-syncer-5llzd\" (UID: \"70f3faf3-b150-468f-abb7-40f2d221164e\") " pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:19.409819 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:19.409789    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:19.409819 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:19.409816    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:19.409884 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:19.409829    2565 projected.go:196] Error preparing data for projected volume kube-api-access-p4tqx for pod openshift-network-diagnostics/network-check-target-zl97d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:19.409934 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:19.409882    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:19.409934 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:19.409893    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx podName:c0a61d36-0e50-4539-ac3f-d5b6c9c3e287 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:20.409874058 +0000 UTC m=+5.975985771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4tqx" (UniqueName: "kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx") pod "network-check-target-zl97d" (UID: "c0a61d36-0e50-4539-ac3f-d5b6c9c3e287") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:19.410022 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:19.409936    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret podName:70f3faf3-b150-468f-abb7-40f2d221164e nodeName:}" failed. No retries permitted until 2026-09-17 16:42:20.409925167 +0000 UTC m=+5.976036881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret") pod "global-pull-secret-syncer-5llzd" (UID: "70f3faf3-b150-468f-abb7-40f2d221164e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:19.730784 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.730742    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 16:37:18 +0000 UTC" deadline="2028-03-05 07:44:47.478690864 +0000 UTC"
Sep 17 16:42:19.730784 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.730778    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12831h2m27.747916609s"
Sep 17 16:42:19.818784 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.818756    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:19.818942 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:19.818876    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:19.833070 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.833021    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvxw9" event={"ID":"c4b8badb-da9b-4399-96ad-aa05263c9542","Type":"ContainerStarted","Data":"931a31f689066c9ee62414069e9e0641b7fa3b2d116cf523ba58cc20a85042b6"}
Sep 17 16:42:19.834636 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.834563    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6hz2q" event={"ID":"8a6a6976-08a4-4f97-9abe-a558269301a0","Type":"ContainerStarted","Data":"626f5ca369ae2c33c82c59e853b53fd3b5957092476a37b85cdbfb00cea21942"}
Sep 17 16:42:19.836589 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.836557    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-69bv6" event={"ID":"307aa194-969e-445a-8f44-22359d43a8b5","Type":"ContainerStarted","Data":"cdd82e1b770468492f855bcb7a5aabf39f226b80f630ec6578c97616d658fac0"}
Sep 17 16:42:19.838381 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.838350    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cqx8k" event={"ID":"ffd47162-fa3a-4c51-a370-3ffe004cf33a","Type":"ContainerStarted","Data":"d20f3acb23af6e9eff08e0b6b312cd81d02227baf75a3926c9ec00f5ca8e29a0"}
Sep 17 16:42:19.839667 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.839641    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s5lfc" event={"ID":"aef9115b-1284-4e79-a8e3-9e8d1778a76e","Type":"ContainerStarted","Data":"006ba8e02eb82d7d1aa801f853c3821c9754e9ce7713204a0f7fb370aae39e5c"}
Sep 17 16:42:19.841205 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.841174    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bn474" event={"ID":"3ce775b0-037d-4846-a3b6-f8bf2f76f293","Type":"ContainerStarted","Data":"9b71eaa2afd0b90b4a6bc72a8487c83c50bf3ef59dfc24d3e967c1ffba583346"}
Sep 17 16:42:19.842583 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.842546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dtmtt" event={"ID":"4a45b30c-1f89-4643-84b2-189a36d6e0f8","Type":"ContainerStarted","Data":"272bb546f66e1bea0e07323e0e6abc9243877e4300f307db171a663220aaeeda"}
Sep 17 16:42:19.843924 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.843885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss" event={"ID":"72016799-7f64-4649-b19d-28618010a6a3","Type":"ContainerStarted","Data":"c83ec1df71ea7c723a88cae247c7ae03c1e1d0e969ab1631858158be1d931822"}
Sep 17 16:42:19.845657 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:19.845534    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29" event={"ID":"700d8b01-1d3b-4b49-bb6a-696687e029bb","Type":"ContainerStarted","Data":"72b45b134a7922d0eb688c924e4d57cf15896eea5d3a8b3766eef1d360986390"}
Sep 17 16:42:20.316653 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:20.316618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs\") pod \"network-metrics-daemon-wld7n\" (UID: \"716feac9-5ed2-4b28-9ede-a210a4871e3d\") " pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:20.316821 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:20.316749    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:20.316821 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:20.316812    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs podName:716feac9-5ed2-4b28-9ede-a210a4871e3d nodeName:}" failed. No retries permitted until 2026-09-17 16:42:22.31679585 +0000 UTC m=+7.882907550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs") pod "network-metrics-daemon-wld7n" (UID: "716feac9-5ed2-4b28-9ede-a210a4871e3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:20.417284 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:20.417253    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tqx\" (UniqueName: \"kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx\") pod \"network-check-target-zl97d\" (UID: \"c0a61d36-0e50-4539-ac3f-d5b6c9c3e287\") " pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:20.417459 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:20.417293    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret\") pod \"global-pull-secret-syncer-5llzd\" (UID: \"70f3faf3-b150-468f-abb7-40f2d221164e\") " pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:20.417459 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:20.417391    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:20.417459 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:20.417435    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret podName:70f3faf3-b150-468f-abb7-40f2d221164e nodeName:}" failed. No retries permitted until 2026-09-17 16:42:22.417422958 +0000 UTC m=+7.983534671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret") pod "global-pull-secret-syncer-5llzd" (UID: "70f3faf3-b150-468f-abb7-40f2d221164e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:20.417729 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:20.417714    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:20.417784 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:20.417733    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:20.417784 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:20.417752    2565 projected.go:196] Error preparing data for projected volume kube-api-access-p4tqx for pod openshift-network-diagnostics/network-check-target-zl97d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:20.417784 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:20.417782    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx podName:c0a61d36-0e50-4539-ac3f-d5b6c9c3e287 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:22.417772813 +0000 UTC m=+7.983884526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4tqx" (UniqueName: "kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx") pod "network-check-target-zl97d" (UID: "c0a61d36-0e50-4539-ac3f-d5b6c9c3e287") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:20.731365 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:20.731323    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 16:37:18 +0000 UTC" deadline="2028-06-14 15:53:45.023941554 +0000 UTC"
Sep 17 16:42:20.731365 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:20.731363    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15263h11m24.292583129s"
Sep 17 16:42:20.820553 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:20.820033    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:20.820553 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:20.820087    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:20.820553 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:20.820185    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:20.820553 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:20.820492    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:21.819582 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:21.819086    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:21.819582 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:21.819206    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:22.331606 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:22.330994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs\") pod \"network-metrics-daemon-wld7n\" (UID: \"716feac9-5ed2-4b28-9ede-a210a4871e3d\") " pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:22.331606 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:22.331145    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:22.331606 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:22.331204    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs podName:716feac9-5ed2-4b28-9ede-a210a4871e3d nodeName:}" failed. No retries permitted until 2026-09-17 16:42:26.331185571 +0000 UTC m=+11.897297277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs") pod "network-metrics-daemon-wld7n" (UID: "716feac9-5ed2-4b28-9ede-a210a4871e3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:22.432173 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:22.431364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tqx\" (UniqueName: \"kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx\") pod \"network-check-target-zl97d\" (UID: \"c0a61d36-0e50-4539-ac3f-d5b6c9c3e287\") " pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:22.432173 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:22.431412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret\") pod \"global-pull-secret-syncer-5llzd\" (UID: \"70f3faf3-b150-468f-abb7-40f2d221164e\") " pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:22.432173 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:22.431530    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:22.432173 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:22.431585    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret podName:70f3faf3-b150-468f-abb7-40f2d221164e nodeName:}" failed. No retries permitted until 2026-09-17 16:42:26.431566961 +0000 UTC m=+11.997678664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret") pod "global-pull-secret-syncer-5llzd" (UID: "70f3faf3-b150-468f-abb7-40f2d221164e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:22.432173 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:22.432016    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:22.432173 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:22.432035    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:22.432173 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:22.432048    2565 projected.go:196] Error preparing data for projected volume kube-api-access-p4tqx for pod openshift-network-diagnostics/network-check-target-zl97d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:22.432173 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:22.432103    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx podName:c0a61d36-0e50-4539-ac3f-d5b6c9c3e287 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:26.432088467 +0000 UTC m=+11.998200168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4tqx" (UniqueName: "kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx") pod "network-check-target-zl97d" (UID: "c0a61d36-0e50-4539-ac3f-d5b6c9c3e287") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:22.818305 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:22.818274    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:22.818513 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:22.818287    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:22.818513 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:22.818416    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:22.818513 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:22.818492    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:23.819224 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:23.819192    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:23.819689 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:23.819332    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:24.819099 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:24.819048    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:24.819267 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:24.819110    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:24.819267 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:24.819207    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:24.819666 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:24.819375    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:25.819082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:25.819002    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:25.819256 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:25.819169    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:26.365344 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:26.365308    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs\") pod \"network-metrics-daemon-wld7n\" (UID: \"716feac9-5ed2-4b28-9ede-a210a4871e3d\") " pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:26.365807 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:26.365480    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:26.365807 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:26.365543    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs podName:716feac9-5ed2-4b28-9ede-a210a4871e3d nodeName:}" failed. No retries permitted until 2026-09-17 16:42:34.365524341 +0000 UTC m=+19.931636045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs") pod "network-metrics-daemon-wld7n" (UID: "716feac9-5ed2-4b28-9ede-a210a4871e3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:26.466434 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:26.466399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tqx\" (UniqueName: \"kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx\") pod \"network-check-target-zl97d\" (UID: \"c0a61d36-0e50-4539-ac3f-d5b6c9c3e287\") " pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:26.466591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:26.466447    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret\") pod \"global-pull-secret-syncer-5llzd\" (UID: \"70f3faf3-b150-468f-abb7-40f2d221164e\") " pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:26.466591 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:26.466574    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:26.466711 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:26.466585    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:26.466711 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:26.466608    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:26.466711 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:26.466618    2565 projected.go:196] Error preparing data for projected volume kube-api-access-p4tqx for pod openshift-network-diagnostics/network-check-target-zl97d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:26.466711 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:26.466631    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret podName:70f3faf3-b150-468f-abb7-40f2d221164e nodeName:}" failed. No retries permitted until 2026-09-17 16:42:34.466613249 +0000 UTC m=+20.032724949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret") pod "global-pull-secret-syncer-5llzd" (UID: "70f3faf3-b150-468f-abb7-40f2d221164e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:26.466711 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:26.466668    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx podName:c0a61d36-0e50-4539-ac3f-d5b6c9c3e287 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:34.466654821 +0000 UTC m=+20.032766534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4tqx" (UniqueName: "kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx") pod "network-check-target-zl97d" (UID: "c0a61d36-0e50-4539-ac3f-d5b6c9c3e287") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:26.819228 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:26.819179    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:26.820089 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:26.820025    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:26.820217 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:26.820187    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:26.820323 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:26.820283    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:27.818933 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:27.818815    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:27.819510 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:27.818963    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:28.818702 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:28.818650    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:28.818702 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:28.818682    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:28.818891 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:28.818803    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:28.818962 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:28.818902    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:29.818415 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:29.818389    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:29.818672 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:29.818490    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:30.818324 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:30.818283    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:30.818784 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:30.818401    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:30.818784 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:30.818462    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:30.818784 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:30.818587    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:31.818680 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:31.818642    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:31.819178 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:31.818767    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:32.819074 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:32.819044    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:32.819454 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:32.819180    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:32.819454 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:32.819241    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:32.819454 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:32.819355    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:33.818870 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:33.818834    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:33.819151 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:33.818980    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:34.421555 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:34.421521    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs\") pod \"network-metrics-daemon-wld7n\" (UID: \"716feac9-5ed2-4b28-9ede-a210a4871e3d\") " pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:34.421716 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:34.421681    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:34.421783 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:34.421751    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs podName:716feac9-5ed2-4b28-9ede-a210a4871e3d nodeName:}" failed. No retries permitted until 2026-09-17 16:42:50.421730554 +0000 UTC m=+35.987842254 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs") pod "network-metrics-daemon-wld7n" (UID: "716feac9-5ed2-4b28-9ede-a210a4871e3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:34.522226 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:34.522192    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tqx\" (UniqueName: \"kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx\") pod \"network-check-target-zl97d\" (UID: \"c0a61d36-0e50-4539-ac3f-d5b6c9c3e287\") " pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:34.522376 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:34.522239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret\") pod \"global-pull-secret-syncer-5llzd\" (UID: \"70f3faf3-b150-468f-abb7-40f2d221164e\") " pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:34.522376 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:34.522358    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:34.522376 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:34.522360    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:34.522486 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:34.522382    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:34.522486 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:34.522392    2565 projected.go:196] Error preparing data for projected volume kube-api-access-p4tqx for pod openshift-network-diagnostics/network-check-target-zl97d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:34.522486 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:34.522412    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret podName:70f3faf3-b150-468f-abb7-40f2d221164e nodeName:}" failed. No retries permitted until 2026-09-17 16:42:50.52239719 +0000 UTC m=+36.088508891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret") pod "global-pull-secret-syncer-5llzd" (UID: "70f3faf3-b150-468f-abb7-40f2d221164e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:34.522486 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:34.522430    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx podName:c0a61d36-0e50-4539-ac3f-d5b6c9c3e287 nodeName:}" failed. No retries permitted until 2026-09-17 16:42:50.522418456 +0000 UTC m=+36.088530170 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4tqx" (UniqueName: "kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx") pod "network-check-target-zl97d" (UID: "c0a61d36-0e50-4539-ac3f-d5b6c9c3e287") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:34.818881 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:34.818801    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:34.819054 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:34.818951    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:34.819054 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:34.818990    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:34.819172 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:34.819077    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:35.818480 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:35.818446    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:35.818639 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:35.818560    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:36.819792 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:36.819668    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:36.820070 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:36.819747    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:36.820070 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:36.819866    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:36.820070 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:36.819988    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:36.882967 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:36.882941    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cqx8k" event={"ID":"ffd47162-fa3a-4c51-a370-3ffe004cf33a","Type":"ContainerStarted","Data":"69b05cc27dc80b3378707599eb0c38aba885644aae09aedd4adec9bf22217847"}
Sep 17 16:42:36.884681 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:36.884655    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s5lfc" event={"ID":"aef9115b-1284-4e79-a8e3-9e8d1778a76e","Type":"ContainerStarted","Data":"03271585c8247ad08a697a2be105cd4836e1f840996cec2519b950aac2c2b257"}
Sep 17 16:42:36.886045 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:36.886024    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal" event={"ID":"b8bce5ac69a7a3fe95f9655d41fae9d1","Type":"ContainerStarted","Data":"a5eb0b96bee15f205ec339bdb8504ef749a3e563a6ef250156a434ee4973288c"}
Sep 17 16:42:36.903173 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:36.903131    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cqx8k" podStartSLOduration=3.312330195 podStartE2EDuration="20.903118587s" totalImagesPullingTime="17.590788392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:36.902988975 +0000 UTC m=+22.469100697" watchObservedRunningTime="2026-09-17 16:42:36.903118587 +0000 UTC m=+22.469230309"
Sep 17 16:42:36.917649 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:36.917608    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal" podStartSLOduration=19.917592874 podStartE2EDuration="19.917592874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:36.917288786 +0000 UTC m=+22.483400509" watchObservedRunningTime="2026-09-17 16:42:36.917592874 +0000 UTC m=+22.483704596"
Sep 17 16:42:36.934578 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:36.934543    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s5lfc" podStartSLOduration=3.325669889 podStartE2EDuration="20.934532078s" totalImagesPullingTime="17.608862189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:36.934166127 +0000 UTC m=+22.500277848" watchObservedRunningTime="2026-09-17 16:42:36.934532078 +0000 UTC m=+22.500643801"
Sep 17 16:42:37.818927 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.818701    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:37.819052 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:37.819028    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:37.889718 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.889682    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-69bv6" event={"ID":"307aa194-969e-445a-8f44-22359d43a8b5","Type":"ContainerStarted","Data":"7f37cd7b00433f343d034abdde074566d17afc4a95827c9b62be8e6b4445828a"}
Sep 17 16:42:37.891253 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.891225    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bn474" event={"ID":"3ce775b0-037d-4846-a3b6-f8bf2f76f293","Type":"ContainerStarted","Data":"9a1d4c12e52a2d91470a1e96b065a4dfe8268519082e5bd2b348ed9e417ff5bc"}
Sep 17 16:42:37.892734 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.892712    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dtmtt" event={"ID":"4a45b30c-1f89-4643-84b2-189a36d6e0f8","Type":"ContainerStarted","Data":"e185adc78d003b713b4370750e629876301e4b0ec1a30aa738dcaf81f364ba27"}
Sep 17 16:42:37.894229 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.894207    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss" event={"ID":"72016799-7f64-4649-b19d-28618010a6a3","Type":"ContainerStarted","Data":"b4b4cf0ed48ab36455788504e28603818ef7982cb8ae48def627951cccee5166"}
Sep 17 16:42:37.897520 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.897498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29" event={"ID":"700d8b01-1d3b-4b49-bb6a-696687e029bb","Type":"ContainerStarted","Data":"a1bdc9320875cceac0267e58d3c45ac1214fa5718c08db128bf63a08a27f70c0"}
Sep 17 16:42:37.897612 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.897568    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29" event={"ID":"700d8b01-1d3b-4b49-bb6a-696687e029bb","Type":"ContainerStarted","Data":"c8582d28bdec7150bfe0f7e96a9f03d722ae22b750e30c547c940bbbe46d9458"}
Sep 17 16:42:37.897668 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.897621    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29" event={"ID":"700d8b01-1d3b-4b49-bb6a-696687e029bb","Type":"ContainerStarted","Data":"de1baf8260ad341be7d80d708ef9efaa5706d0bdf8aa7c34be7406518afff0b6"}
Sep 17 16:42:37.897732 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.897671    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29" event={"ID":"700d8b01-1d3b-4b49-bb6a-696687e029bb","Type":"ContainerStarted","Data":"a467c88a93088fca1fb8594e1bca673b2f4de782e127f95a180db3b123231a3a"}
Sep 17 16:42:37.897732 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.897718    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29" event={"ID":"700d8b01-1d3b-4b49-bb6a-696687e029bb","Type":"ContainerStarted","Data":"fe1ca0fb3e0b4e1f5f82067ad1d1db2c8d6c76cef4beac961f59a84acfc63c57"}
Sep 17 16:42:37.897825 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.897766    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29" event={"ID":"700d8b01-1d3b-4b49-bb6a-696687e029bb","Type":"ContainerStarted","Data":"8d4d451ec712b608099657e4e411c17bd655e5b38e5e5d07bf0744dd70cba04f"}
Sep 17 16:42:37.899045 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.899021    2565 generic.go:356] "Generic (PLEG): container finished" podID="b6d096ee74184de72596674c90aed240" containerID="2bbb0687a830056d6f76899dd0571fe228a914349ca364235d533cf22b9719cf" exitCode=0
Sep 17 16:42:37.899132 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.899101    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal" event={"ID":"b6d096ee74184de72596674c90aed240","Type":"ContainerDied","Data":"2bbb0687a830056d6f76899dd0571fe228a914349ca364235d533cf22b9719cf"}
Sep 17 16:42:37.900744 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.900723    2565 generic.go:356] "Generic (PLEG): container finished" podID="c4b8badb-da9b-4399-96ad-aa05263c9542" containerID="d43fb64e731c03d28b71ff5aea0b2d8903578356e69936ba30bde43a7e04bb9c" exitCode=0
Sep 17 16:42:37.900826 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.900803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvxw9" event={"ID":"c4b8badb-da9b-4399-96ad-aa05263c9542","Type":"ContainerDied","Data":"d43fb64e731c03d28b71ff5aea0b2d8903578356e69936ba30bde43a7e04bb9c"}
Sep 17 16:42:37.905716 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.905656    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-69bv6" podStartSLOduration=4.731218762 podStartE2EDuration="21.905645374s" totalImagesPullingTime="17.174426612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:37.905152773 +0000 UTC m=+23.471264487" watchObservedRunningTime="2026-09-17 16:42:37.905645374 +0000 UTC m=+23.471757139"
Sep 17 16:42:37.922397 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.922353    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dtmtt" podStartSLOduration=4.296896952 podStartE2EDuration="21.92233735s" totalImagesPullingTime="17.625440398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:37.921497926 +0000 UTC m=+23.487609649" watchObservedRunningTime="2026-09-17 16:42:37.92233735 +0000 UTC m=+23.488449077"
Sep 17 16:42:37.963114 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.963075    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bn474" podStartSLOduration=4.352594705 podStartE2EDuration="21.963058487s" totalImagesPullingTime="17.610463782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:37.962179323 +0000 UTC m=+23.528291042" watchObservedRunningTime="2026-09-17 16:42:37.963058487 +0000 UTC m=+23.529170201"
Sep 17 16:42:37.981612 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:37.981589    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:38.080227 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:38.080200    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:38.144375 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:38.144193    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 16:42:38.541799 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:38.541708    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:38.605097 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:38.605061    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:38.720016 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:38.719952    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:38.776339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:38.776235    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T16:42:38.144370786Z","UUID":"c02e33cc-3109-4f22-bf23-0f3a4e3a2b11","Handler":null,"Name":"","Endpoint":""}
Sep 17 16:42:38.777906 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:38.777881    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 16:42:38.778042 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:38.777934    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 16:42:38.818540 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:38.818398    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:38.818540 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:38.818514    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:38.818728 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:38.818545    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:38.818728 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:38.818645    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:38.819878 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:38.819853    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:38.907252 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:38.907221    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss" event={"ID":"72016799-7f64-4649-b19d-28618010a6a3","Type":"ContainerStarted","Data":"414b85858f54c05286626d3ce26073d607bf02351c62c97a2d5994e4484a5922"}
Sep 17 16:42:38.910503 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:38.910458    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal" event={"ID":"b6d096ee74184de72596674c90aed240","Type":"ContainerStarted","Data":"bda4906b3806aa1372070e2a27b5176fc8be84af52d811ab3a295ac9864a8686"}
Sep 17 16:42:38.913949 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:38.913366    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6hz2q" event={"ID":"8a6a6976-08a4-4f97-9abe-a558269301a0","Type":"ContainerStarted","Data":"87786030ad2425c1364eb68d34498149571f2b876e686e2b92fb8e9bd089bd1e"}
Sep 17 16:42:38.926971 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:38.926929    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal" podStartSLOduration=21.926892411 podStartE2EDuration="21.926892411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:38.926624309 +0000 UTC m=+24.492736033" watchObservedRunningTime="2026-09-17 16:42:38.926892411 +0000 UTC m=+24.493004134"
Sep 17 16:42:38.944422 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:38.944380    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6hz2q" podStartSLOduration=5.36836087 podStartE2EDuration="22.944366876s" totalImagesPullingTime="17.576006006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:38.943779953 +0000 UTC m=+24.509891678" watchObservedRunningTime="2026-09-17 16:42:38.944366876 +0000 UTC m=+24.510478598"
Sep 17 16:42:39.149281 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:39.149245    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:39.249552 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:39.249523    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:39.654920 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:39.654883    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bn474"
Sep 17 16:42:39.655418 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:39.655398    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bn474"
Sep 17 16:42:39.700215 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:39.700168    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:39.818981 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:39.818928    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:39.819125 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:39.819105    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:39.917659 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:39.917624    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss" event={"ID":"72016799-7f64-4649-b19d-28618010a6a3","Type":"ContainerStarted","Data":"372eede9488ef9998295ecced99911ebe97ae69dfd71ba8b76513d23709e9c2b"}
Sep 17 16:42:39.921656 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:39.921591    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29" event={"ID":"700d8b01-1d3b-4b49-bb6a-696687e029bb","Type":"ContainerStarted","Data":"0b366bfd45d17ae74db1cde8840b11f192c22cdfcf1479d688007dfdfdbee551"}
Sep 17 16:42:39.922183 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:39.921947    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bn474"
Sep 17 16:42:39.922445 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:39.922423    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bn474"
Sep 17 16:42:39.935895 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:39.935856    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pfqss" podStartSLOduration=8.74604179 podStartE2EDuration="23.935803808s" totalImagesPullingTime="15.189762018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:39.935678032 +0000 UTC m=+25.501789755" watchObservedRunningTime="2026-09-17 16:42:39.935803808 +0000 UTC m=+25.501915532"
Sep 17 16:42:40.503163 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:40.503128    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:40.603132 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:40.603100    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:40.703622 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:40.703593    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:40.804543 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:40.804458    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:40.818730 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:40.818706    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:40.818869 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:40.818826    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:40.818961 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:40.818875    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:40.819022 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:40.818991    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:40.907643 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:40.907614    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:41.204637 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:41.204599    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:41.818436 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:41.818399    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:41.818570 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:41.818513    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:41.930112 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:41.929794    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29" event={"ID":"700d8b01-1d3b-4b49-bb6a-696687e029bb","Type":"ContainerStarted","Data":"9940b7748433adf6d5645965d9d81ac2bd843533983d355b102f0ece16f650f8"}
Sep 17 16:42:41.930112 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:41.930014    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:41.930112 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:41.930084    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:41.930360 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:41.930121    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:41.946435 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:41.946262    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:41.946435 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:41.946378    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:42:41.972623 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:41.972567    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29" podStartSLOduration=8.055418527 podStartE2EDuration="25.972546602s" totalImagesPullingTime="17.917128075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:41.970800129 +0000 UTC m=+27.536911851" watchObservedRunningTime="2026-09-17 16:42:41.972546602 +0000 UTC m=+27.538658326"
Sep 17 16:42:42.210115 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:42.209948    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:42.394094 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:42.394066    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:42.793768 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:42.793709    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:42.818778 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:42.818751    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:42.818904 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:42.818884    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:42.818997 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:42.818931    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:42.819056 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:42.819011    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:42.933084 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:42.933059    2565 generic.go:356] "Generic (PLEG): container finished" podID="c4b8badb-da9b-4399-96ad-aa05263c9542" containerID="8345f4d51743a7fc98dd2d621034bcf2f91b44de82960fb49b983e5af0fa0916" exitCode=0
Sep 17 16:42:42.933188 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:42.933087    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvxw9" event={"ID":"c4b8badb-da9b-4399-96ad-aa05263c9542","Type":"ContainerDied","Data":"8345f4d51743a7fc98dd2d621034bcf2f91b44de82960fb49b983e5af0fa0916"}
Sep 17 16:42:43.001907 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:43.001885    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:43.404258 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:43.404228    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:43.504194 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:43.504173    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:43.613518 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:43.613493    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:43.734117 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:43.734068    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:43.819177 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:43.819156    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:43.819276 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:43.819240    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:44.196559 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:44.196535    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:44.387216 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:44.387191    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:44.451459 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:44.451437    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:44.818834 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:44.818790    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:44.818934 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:44.818790    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:44.818934 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:44.818885    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:44.819022 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:44.818966    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:44.830599 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:44.830580    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:44.938993 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:44.938973    2565 generic.go:356] "Generic (PLEG): container finished" podID="c4b8badb-da9b-4399-96ad-aa05263c9542" containerID="4ed1a9f57fb2ff8903e0dedaa6182bcbec812f3859378ff0b13c66e7313879c7" exitCode=0
Sep 17 16:42:44.939086 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:44.939004    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvxw9" event={"ID":"c4b8badb-da9b-4399-96ad-aa05263c9542","Type":"ContainerDied","Data":"4ed1a9f57fb2ff8903e0dedaa6182bcbec812f3859378ff0b13c66e7313879c7"}
Sep 17 16:42:44.993222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:44.993199    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:45.094069 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:45.094022    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:45.599083 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:45.599051    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:45.818496 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:45.818470    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:45.818628 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:45.818555    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:46.194504 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:46.194480    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:46.401271 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:46.401238    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:46.596421 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:46.596391    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:46.796973 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:46.796903    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:46.819155 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:46.819134    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:46.819267 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:46.819248    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:46.819350 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:46.819330    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:46.819464 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:46.819442    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:46.945250 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:46.945227    2565 generic.go:356] "Generic (PLEG): container finished" podID="c4b8badb-da9b-4399-96ad-aa05263c9542" containerID="4079c72027d0e7eb4a21eb09a9dbc26196ecfef38dbbee2119165263c484c91d" exitCode=0
Sep 17 16:42:46.945349 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:46.945267    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvxw9" event={"ID":"c4b8badb-da9b-4399-96ad-aa05263c9542","Type":"ContainerDied","Data":"4079c72027d0e7eb4a21eb09a9dbc26196ecfef38dbbee2119165263c484c91d"}
Sep 17 16:42:46.985465 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:46.985441    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:47.394861 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:47.394834    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:47.560095 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:47.560066    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:47.795632 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:47.795561    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:47.818604 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:47.818576    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:47.818974 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:47.818680    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:48.393480 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:48.393446    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:48.793116 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:48.793041    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:48.818224 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:48.818201    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:48.818384 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:48.818318    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:48.818464 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:48.818431    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:48.818567 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:48.818543    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:48.997127 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:48.997100    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:49.195667 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:49.195643    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:49.818540 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:49.818506    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:49.818711 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:49.818609    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:49.934801 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:49.934768    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:50.035606 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:50.035568    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:50.399734 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:50.399705    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:50.447339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:50.447311    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs\") pod \"network-metrics-daemon-wld7n\" (UID: \"716feac9-5ed2-4b28-9ede-a210a4871e3d\") " pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:50.447470 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:50.447440    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:50.447515 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:50.447496    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs podName:716feac9-5ed2-4b28-9ede-a210a4871e3d nodeName:}" failed. No retries permitted until 2026-09-17 16:43:22.447482576 +0000 UTC m=+68.013594291 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs") pod "network-metrics-daemon-wld7n" (UID: "716feac9-5ed2-4b28-9ede-a210a4871e3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 16:42:50.548459 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:50.548433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tqx\" (UniqueName: \"kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx\") pod \"network-check-target-zl97d\" (UID: \"c0a61d36-0e50-4539-ac3f-d5b6c9c3e287\") " pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:50.548615 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:50.548469    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret\") pod \"global-pull-secret-syncer-5llzd\" (UID: \"70f3faf3-b150-468f-abb7-40f2d221164e\") " pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:50.548615 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:50.548592    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:50.548731 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:50.548614    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 16:42:50.548731 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:50.548639    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 16:42:50.548731 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:50.548652    2565 projected.go:196] Error preparing data for projected volume kube-api-access-p4tqx for pod openshift-network-diagnostics/network-check-target-zl97d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:50.548731 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:50.548653    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret podName:70f3faf3-b150-468f-abb7-40f2d221164e nodeName:}" failed. No retries permitted until 2026-09-17 16:43:22.548635755 +0000 UTC m=+68.114747460 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret") pod "global-pull-secret-syncer-5llzd" (UID: "70f3faf3-b150-468f-abb7-40f2d221164e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 16:42:50.548731 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:50.548725    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx podName:c0a61d36-0e50-4539-ac3f-d5b6c9c3e287 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:22.54870895 +0000 UTC m=+68.114820666 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4tqx" (UniqueName: "kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx") pod "network-check-target-zl97d" (UID: "c0a61d36-0e50-4539-ac3f-d5b6c9c3e287") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 16:42:50.800838 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:50.800756    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:50.818754 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:50.818726    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:50.818906 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:50.818842    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:50.818980 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:50.818940    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:50.819082 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:50.819059    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:51.011162 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:51.011125    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:51.192698 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:51.192671    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:51.690780 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:51.690753    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:51.818749 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:51.818704    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:51.818833 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:51.818781    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:51.959395 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:51.959368    2565 generic.go:356] "Generic (PLEG): container finished" podID="c4b8badb-da9b-4399-96ad-aa05263c9542" containerID="edf4b7f43d3513662705bb20a87763ed5fa650a9b9614176b3a35931cffe656e" exitCode=0
Sep 17 16:42:51.959529 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:51.959406    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvxw9" event={"ID":"c4b8badb-da9b-4399-96ad-aa05263c9542","Type":"ContainerDied","Data":"edf4b7f43d3513662705bb20a87763ed5fa650a9b9614176b3a35931cffe656e"}
Sep 17 16:42:52.818557 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:52.818532    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:52.818857 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:52.818535    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:52.818857 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:52.818666    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:52.818857 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:52.818705    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:52.963818 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:52.963794    2565 generic.go:356] "Generic (PLEG): container finished" podID="c4b8badb-da9b-4399-96ad-aa05263c9542" containerID="09fb908acdc0d1fcd560c14dece5416411f2370edc460a2de8ef8c02738b2d9b" exitCode=0
Sep 17 16:42:52.963935 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:52.963826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvxw9" event={"ID":"c4b8badb-da9b-4399-96ad-aa05263c9542","Type":"ContainerDied","Data":"09fb908acdc0d1fcd560c14dece5416411f2370edc460a2de8ef8c02738b2d9b"}
Sep 17 16:42:53.819135 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:53.819099    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:53.819496 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:53.819201    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:53.969727 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:53.969690    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qvxw9" event={"ID":"c4b8badb-da9b-4399-96ad-aa05263c9542","Type":"ContainerStarted","Data":"b4d8f07ba7e8505fb6720463e2f89b51ff331c26f450c05dface0ba597b10330"}
Sep 17 16:42:53.998293 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:53.998249    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qvxw9" podStartSLOduration=9.675976697 podStartE2EDuration="37.998234851s" totalImagesPullingTime="28.322258154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:53.997935344 +0000 UTC m=+39.564047060" watchObservedRunningTime="2026-09-17 16:42:53.998234851 +0000 UTC m=+39.564346573"
Sep 17 16:42:54.663626 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:54.663574    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:54.741041 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:54.741011    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:54.817392 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:54.817364    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:54.818339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:54.818321    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:54.818455 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:54.818434    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:54.818495 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:54.818486    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:54.818590 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:54.818571    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:55.113194 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:55.113176    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:55.664757 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:55.664729    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:55.797791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:55.797769    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:55.818930 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:55.818890    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:55.819020 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:55.818991    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:56.819624 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:56.819589    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:56.820039 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:56.819701    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:56.820039 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:56.819723    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:56.820039 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:56.819784    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:57.016518 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:57.016342    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:57.116487 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:57.116465    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:57.186610 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:57.186590    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:57.281135 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:57.281114    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:57.456439 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:57.456370    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:57.557135 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:57.557109    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:57.661247 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:57.661226    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:57.735375 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:57.735315    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:57.819026 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:57.819006    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:57.819110 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:57.819097    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:58.818408 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:58.818388    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:42:58.818708 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:58.818478    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:42:58.818708 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:58.818554    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:42:58.818708 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:58.818629    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:42:58.955907 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:58.955887    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:59.173343 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:59.173323    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:59.270103 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:59.270082    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:59.368135 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:59.368113    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:59.466800 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:59.466754    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:59.579983 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:59.579962    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:59.666205 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:59.666179    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:42:59.818223 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:59.818182    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:42:59.818293 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:42:59.818260    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:42:59.922202 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:42:59.922181    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:00.021663 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:00.021644    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:00.796013 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:00.795984    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zl97d"]
Sep 17 16:43:00.796145 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:00.796092    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:43:00.796238 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:43:00.796205    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:43:00.800545 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:00.800095    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wld7n"]
Sep 17 16:43:00.800545 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:00.800210    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:43:00.800545 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:43:00.800369    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:43:00.801536 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:00.801502    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5llzd"]
Sep 17 16:43:00.801772 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:00.801754    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:43:00.802009 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:43:00.801985    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:43:01.866123 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:01.866090    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:01.966533 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:01.966493    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:02.170963 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:02.170894    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:02.818828 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:02.818798    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:43:02.819032 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:02.818802    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:43:02.819032 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:43:02.818930    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:43:02.819152 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:43:02.819052    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:43:02.819152 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:02.819102    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:43:02.819256 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:43:02.819212    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:43:04.819155 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:04.819122    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:43:04.819537 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:04.819122    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:43:04.819537 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:43:04.819223    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zl97d" podUID="c0a61d36-0e50-4539-ac3f-d5b6c9c3e287"
Sep 17 16:43:04.819537 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:43:04.819267    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5llzd" podUID="70f3faf3-b150-468f-abb7-40f2d221164e"
Sep 17 16:43:04.819537 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:04.819310    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:43:04.819537 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:43:04.819406    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wld7n" podUID="716feac9-5ed2-4b28-9ede-a210a4871e3d"
Sep 17 16:43:05.753561 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.753543    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeReady"
Sep 17 16:43:05.753665 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.753621    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 16:43:05.802607 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.802584    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7h2vr"]
Sep 17 16:43:05.804317 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.804303    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8l5mv"]
Sep 17 16:43:05.804435 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.804418    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7h2vr"
Sep 17 16:43:05.805857 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.805838    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:05.807142 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.807120    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 16:43:05.807234 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.807153    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tcspl\""
Sep 17 16:43:05.807290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.807252    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 16:43:05.807460 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.807444    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 16:43:05.808700 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.808680    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 16:43:05.808780 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.808705    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 16:43:05.808780 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.808712    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 16:43:05.808892 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.808781    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d57nv\""
Sep 17 16:43:05.808892 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.808794    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 16:43:05.819875 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.819856    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7h2vr"]
Sep 17 16:43:05.824059 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.824033    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8l5mv"]
Sep 17 16:43:05.824624 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.824608    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-28z7f"]
Sep 17 16:43:05.826303 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.826286    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-28z7f"
Sep 17 16:43:05.828775 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.828733    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v5qhp\""
Sep 17 16:43:05.828775 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.828761    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 16:43:05.828928 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.828834    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 16:43:05.844501 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.844478    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-28z7f"]
Sep 17 16:43:05.950539 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.950302    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:05.957337 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.957310    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61ab0f21-2b0c-4e87-9887-8bbc281cf2d5-config-volume\") pod \"dns-default-28z7f\" (UID: \"61ab0f21-2b0c-4e87-9887-8bbc281cf2d5\") " pod="openshift-dns/dns-default-28z7f"
Sep 17 16:43:05.957408 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.957354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7274b4f9-685c-4c23-abb9-4c6f88fcdcc7-crio-socket\") pod \"insights-runtime-extractor-8l5mv\" (UID: \"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7\") " pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:05.957408 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.957373    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ab0f21-2b0c-4e87-9887-8bbc281cf2d5-metrics-tls\") pod \"dns-default-28z7f\" (UID: \"61ab0f21-2b0c-4e87-9887-8bbc281cf2d5\") " pod="openshift-dns/dns-default-28z7f"
Sep 17 16:43:05.957477 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.957410    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7274b4f9-685c-4c23-abb9-4c6f88fcdcc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8l5mv\" (UID: \"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7\") " pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:05.957477 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.957427    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7274b4f9-685c-4c23-abb9-4c6f88fcdcc7-data-volume\") pod \"insights-runtime-extractor-8l5mv\" (UID: \"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7\") " pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:05.957477 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.957441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7274b4f9-685c-4c23-abb9-4c6f88fcdcc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8l5mv\" (UID: \"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7\") " pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:05.957477 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.957454    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a356edd2-b0ce-4622-be5a-eb1062dfef07-cert\") pod \"ingress-canary-7h2vr\" (UID: \"a356edd2-b0ce-4622-be5a-eb1062dfef07\") " pod="openshift-ingress-canary/ingress-canary-7h2vr"
Sep 17 16:43:05.957595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.957507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9k8p\" (UniqueName: \"kubernetes.io/projected/61ab0f21-2b0c-4e87-9887-8bbc281cf2d5-kube-api-access-t9k8p\") pod \"dns-default-28z7f\" (UID: \"61ab0f21-2b0c-4e87-9887-8bbc281cf2d5\") " pod="openshift-dns/dns-default-28z7f"
Sep 17 16:43:05.957595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.957551    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdzkn\" (UniqueName: \"kubernetes.io/projected/a356edd2-b0ce-4622-be5a-eb1062dfef07-kube-api-access-qdzkn\") pod \"ingress-canary-7h2vr\" (UID: \"a356edd2-b0ce-4622-be5a-eb1062dfef07\") " pod="openshift-ingress-canary/ingress-canary-7h2vr"
Sep 17 16:43:05.957595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.957578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7w6d\" (UniqueName: \"kubernetes.io/projected/7274b4f9-685c-4c23-abb9-4c6f88fcdcc7-kube-api-access-k7w6d\") pod \"insights-runtime-extractor-8l5mv\" (UID: \"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7\") " pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:05.957701 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:05.957593    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ab0f21-2b0c-4e87-9887-8bbc281cf2d5-tmp-dir\") pod \"dns-default-28z7f\" (UID: \"61ab0f21-2b0c-4e87-9887-8bbc281cf2d5\") " pod="openshift-dns/dns-default-28z7f"
Sep 17 16:43:06.058626 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.058576    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7w6d\" (UniqueName: \"kubernetes.io/projected/7274b4f9-685c-4c23-abb9-4c6f88fcdcc7-kube-api-access-k7w6d\") pod \"insights-runtime-extractor-8l5mv\" (UID: \"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7\") " pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:06.058626 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.058604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ab0f21-2b0c-4e87-9887-8bbc281cf2d5-tmp-dir\") pod \"dns-default-28z7f\" (UID: \"61ab0f21-2b0c-4e87-9887-8bbc281cf2d5\") " pod="openshift-dns/dns-default-28z7f"
Sep 17 16:43:06.058626 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.058621    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61ab0f21-2b0c-4e87-9887-8bbc281cf2d5-config-volume\") pod \"dns-default-28z7f\" (UID: \"61ab0f21-2b0c-4e87-9887-8bbc281cf2d5\") " pod="openshift-dns/dns-default-28z7f"
Sep 17 16:43:06.058811 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.058639    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7274b4f9-685c-4c23-abb9-4c6f88fcdcc7-crio-socket\") pod \"insights-runtime-extractor-8l5mv\" (UID: \"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7\") " pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:06.058811 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.058657    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ab0f21-2b0c-4e87-9887-8bbc281cf2d5-metrics-tls\") pod \"dns-default-28z7f\" (UID: \"61ab0f21-2b0c-4e87-9887-8bbc281cf2d5\") " pod="openshift-dns/dns-default-28z7f"
Sep 17 16:43:06.058811 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.058720    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7274b4f9-685c-4c23-abb9-4c6f88fcdcc7-crio-socket\") pod \"insights-runtime-extractor-8l5mv\" (UID: \"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7\") " pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:06.058811 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.058759    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7274b4f9-685c-4c23-abb9-4c6f88fcdcc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8l5mv\" (UID: \"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7\") " pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:06.058811 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.058799    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7274b4f9-685c-4c23-abb9-4c6f88fcdcc7-data-volume\") pod \"insights-runtime-extractor-8l5mv\" (UID: \"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7\") " pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:06.059059 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.058825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7274b4f9-685c-4c23-abb9-4c6f88fcdcc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8l5mv\" (UID: \"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7\") " pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:06.059059 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.058908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a356edd2-b0ce-4622-be5a-eb1062dfef07-cert\") pod \"ingress-canary-7h2vr\" (UID: \"a356edd2-b0ce-4622-be5a-eb1062dfef07\") " pod="openshift-ingress-canary/ingress-canary-7h2vr"
Sep 17 16:43:06.059059 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.058954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9k8p\" (UniqueName: \"kubernetes.io/projected/61ab0f21-2b0c-4e87-9887-8bbc281cf2d5-kube-api-access-t9k8p\") pod \"dns-default-28z7f\" (UID: \"61ab0f21-2b0c-4e87-9887-8bbc281cf2d5\") " pod="openshift-dns/dns-default-28z7f"
Sep 17 16:43:06.059059 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.058970    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ab0f21-2b0c-4e87-9887-8bbc281cf2d5-tmp-dir\") pod \"dns-default-28z7f\" (UID: \"61ab0f21-2b0c-4e87-9887-8bbc281cf2d5\") " pod="openshift-dns/dns-default-28z7f"
Sep 17 16:43:06.059059 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.058976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdzkn\" (UniqueName: \"kubernetes.io/projected/a356edd2-b0ce-4622-be5a-eb1062dfef07-kube-api-access-qdzkn\") pod \"ingress-canary-7h2vr\" (UID: \"a356edd2-b0ce-4622-be5a-eb1062dfef07\") " pod="openshift-ingress-canary/ingress-canary-7h2vr"
Sep 17 16:43:06.059262 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.059171    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7274b4f9-685c-4c23-abb9-4c6f88fcdcc7-data-volume\") pod \"insights-runtime-extractor-8l5mv\" (UID: \"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7\") " pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:06.059302 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.059290    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61ab0f21-2b0c-4e87-9887-8bbc281cf2d5-config-volume\") pod \"dns-default-28z7f\" (UID: \"61ab0f21-2b0c-4e87-9887-8bbc281cf2d5\") " pod="openshift-dns/dns-default-28z7f"
Sep 17 16:43:06.059490 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.059470    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7274b4f9-685c-4c23-abb9-4c6f88fcdcc7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8l5mv\" (UID: \"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7\") " pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:06.062998 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.062883    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7274b4f9-685c-4c23-abb9-4c6f88fcdcc7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8l5mv\" (UID: \"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7\") " pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:06.063078 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.062905    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61ab0f21-2b0c-4e87-9887-8bbc281cf2d5-metrics-tls\") pod \"dns-default-28z7f\" (UID: \"61ab0f21-2b0c-4e87-9887-8bbc281cf2d5\") " pod="openshift-dns/dns-default-28z7f"
Sep 17 16:43:06.063532 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.063514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a356edd2-b0ce-4622-be5a-eb1062dfef07-cert\") pod \"ingress-canary-7h2vr\" (UID: \"a356edd2-b0ce-4622-be5a-eb1062dfef07\") " pod="openshift-ingress-canary/ingress-canary-7h2vr"
Sep 17 16:43:06.068286 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.068264    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdzkn\" (UniqueName: \"kubernetes.io/projected/a356edd2-b0ce-4622-be5a-eb1062dfef07-kube-api-access-qdzkn\") pod \"ingress-canary-7h2vr\" (UID: \"a356edd2-b0ce-4622-be5a-eb1062dfef07\") " pod="openshift-ingress-canary/ingress-canary-7h2vr"
Sep 17 16:43:06.068793 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.068774    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7w6d\" (UniqueName: \"kubernetes.io/projected/7274b4f9-685c-4c23-abb9-4c6f88fcdcc7-kube-api-access-k7w6d\") pod \"insights-runtime-extractor-8l5mv\" (UID: \"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7\") " pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:06.070001 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.069977    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9k8p\" (UniqueName: \"kubernetes.io/projected/61ab0f21-2b0c-4e87-9887-8bbc281cf2d5-kube-api-access-t9k8p\") pod \"dns-default-28z7f\" (UID: \"61ab0f21-2b0c-4e87-9887-8bbc281cf2d5\") " pod="openshift-dns/dns-default-28z7f"
Sep 17 16:43:06.114876 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.114858    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7h2vr"
Sep 17 16:43:06.119512 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.119492    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8l5mv"
Sep 17 16:43:06.134207 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.134181    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-28z7f"
Sep 17 16:43:06.160371 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.160333    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:06.399373 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.399175    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7h2vr"]
Sep 17 16:43:06.401844 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:43:06.401816    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda356edd2_b0ce_4622_be5a_eb1062dfef07.slice/crio-e8590c9a6fb7407d1c46d3bc2b8f5f2af9ebce61906bdf885e7f424b7933018a WatchSource:0}: Error finding container e8590c9a6fb7407d1c46d3bc2b8f5f2af9ebce61906bdf885e7f424b7933018a: Status 404 returned error can't find the container with id e8590c9a6fb7407d1c46d3bc2b8f5f2af9ebce61906bdf885e7f424b7933018a
Sep 17 16:43:06.401844 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.401841    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8l5mv"]
Sep 17 16:43:06.404451 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:43:06.404430    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7274b4f9_685c_4c23_abb9_4c6f88fcdcc7.slice/crio-e30a02d85122176dabf7ab957890a376db0a56c8970c2cc864f4f93b9ac2c15a WatchSource:0}: Error finding container e30a02d85122176dabf7ab957890a376db0a56c8970c2cc864f4f93b9ac2c15a: Status 404 returned error can't find the container with id e30a02d85122176dabf7ab957890a376db0a56c8970c2cc864f4f93b9ac2c15a
Sep 17 16:43:06.415542 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.415516    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-28z7f"]
Sep 17 16:43:06.418303 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:43:06.418280    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ab0f21_2b0c_4e87_9887_8bbc281cf2d5.slice/crio-a3b55ce0d69f79c831e6a3d9aae310e1556d1bd1999c39509b3b24d951c43c6f WatchSource:0}: Error finding container a3b55ce0d69f79c831e6a3d9aae310e1556d1bd1999c39509b3b24d951c43c6f: Status 404 returned error can't find the container with id a3b55ce0d69f79c831e6a3d9aae310e1556d1bd1999c39509b3b24d951c43c6f
Sep 17 16:43:06.507202 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.507178    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:06.566167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.566142    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:06.820250 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.820184    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:43:06.820859 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.820276    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:43:06.820859 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.820315    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:43:06.823183 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.823167    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 16:43:06.824149 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.824128    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 16:43:06.824260 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.824159    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 16:43:06.824260 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.824177    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 16:43:06.824353 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.824307    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ls2kp\""
Sep 17 16:43:06.824414 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.824399    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kh76t\""
Sep 17 16:43:06.999886 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.999849    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8l5mv" event={"ID":"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7","Type":"ContainerStarted","Data":"4b2d704ec8efedf872096e103c803c9eeafee75b65632d8d7a56a7b1cd76ac84"}
Sep 17 16:43:07.000057 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:06.999936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8l5mv" event={"ID":"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7","Type":"ContainerStarted","Data":"e30a02d85122176dabf7ab957890a376db0a56c8970c2cc864f4f93b9ac2c15a"}
Sep 17 16:43:07.003970 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:07.003944    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7h2vr" event={"ID":"a356edd2-b0ce-4622-be5a-eb1062dfef07","Type":"ContainerStarted","Data":"e8590c9a6fb7407d1c46d3bc2b8f5f2af9ebce61906bdf885e7f424b7933018a"}
Sep 17 16:43:07.005600 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:07.005574    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-28z7f" event={"ID":"61ab0f21-2b0c-4e87-9887-8bbc281cf2d5","Type":"ContainerStarted","Data":"a3b55ce0d69f79c831e6a3d9aae310e1556d1bd1999c39509b3b24d951c43c6f"}
Sep 17 16:43:08.010340 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:08.010301    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8l5mv" event={"ID":"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7","Type":"ContainerStarted","Data":"a132d164ec3c96cc1f61893ec3001f727abef7b6350f09332c554fca042986a7"}
Sep 17 16:43:08.600237 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:08.600201    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:09.506272 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:09.506247    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:10.009759 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.009725    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:10.020089 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.020049    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8l5mv" event={"ID":"7274b4f9-685c-4c23-abb9-4c6f88fcdcc7","Type":"ContainerStarted","Data":"86390eeedf11cf3fec046c4d7b2f6bd6cbb5a1015c5c84dc97c680883e7c7c38"}
Sep 17 16:43:10.021617 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.021593    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7h2vr" event={"ID":"a356edd2-b0ce-4622-be5a-eb1062dfef07","Type":"ContainerStarted","Data":"004e8f74a451a26474900416f4fb7927919683c46383809b5719b512909d4554"}
Sep 17 16:43:10.023609 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.023583    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-28z7f" event={"ID":"61ab0f21-2b0c-4e87-9887-8bbc281cf2d5","Type":"ContainerStarted","Data":"3430b657369c09c89239cbefc5dc479e96490d59e590a226563effb4956a645d"}
Sep 17 16:43:10.023696 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.023634    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-28z7f" event={"ID":"61ab0f21-2b0c-4e87-9887-8bbc281cf2d5","Type":"ContainerStarted","Data":"48dfbafdf78778e70e3264c4b9338d8b886026ff05ef5b38dbedd56e4e708c41"}
Sep 17 16:43:10.023696 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.023665    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-28z7f"
Sep 17 16:43:10.042048 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.042009    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8l5mv" podStartSLOduration=2.250503468 podStartE2EDuration="5.041996461s" totalImagesPullingTime="2.791492993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:43:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:10.040456741 +0000 UTC m=+55.606568465" watchObservedRunningTime="2026-09-17 16:43:10.041996461 +0000 UTC m=+55.608108179"
Sep 17 16:43:10.060538 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.060499    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-28z7f" podStartSLOduration=2.147541152 podStartE2EDuration="5.060484398s" totalImagesPullingTime="2.912943246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:10.059594511 +0000 UTC m=+55.625706233" watchObservedRunningTime="2026-09-17 16:43:10.060484398 +0000 UTC m=+55.626596139"
Sep 17 16:43:10.077168 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.077125    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7h2vr" podStartSLOduration=2.136898221 podStartE2EDuration="5.077107584s" totalImagesPullingTime="2.940209363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:10.076800958 +0000 UTC m=+55.642912679" watchObservedRunningTime="2026-09-17 16:43:10.077107584 +0000 UTC m=+55.643219304"
Sep 17 16:43:10.185198 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.185172    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:10.265058 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.264996    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"]
Sep 17 16:43:10.268056 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.268021    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"
Sep 17 16:43:10.268428 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.268405    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:10.270551 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.270525    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 16:43:10.270729 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.270715    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 16:43:10.270877 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.270832    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 16:43:10.270975 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.270948    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wxns4\""
Sep 17 16:43:10.271036 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.270984    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 16:43:10.271036 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.270990    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 16:43:10.289439 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.289419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a691796a-00c8-41d7-8440-bb98d19476f5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-k4cjr\" (UID: \"a691796a-00c8-41d7-8440-bb98d19476f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"
Sep 17 16:43:10.289533 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.289446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67zwk\" (UniqueName: \"kubernetes.io/projected/a691796a-00c8-41d7-8440-bb98d19476f5-kube-api-access-67zwk\") pod \"openshift-state-metrics-86c5bc79d6-k4cjr\" (UID: \"a691796a-00c8-41d7-8440-bb98d19476f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"
Sep 17 16:43:10.289533 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.289466    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a691796a-00c8-41d7-8440-bb98d19476f5-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-k4cjr\" (UID: \"a691796a-00c8-41d7-8440-bb98d19476f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"
Sep 17 16:43:10.289638 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.289565    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a691796a-00c8-41d7-8440-bb98d19476f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-k4cjr\" (UID: \"a691796a-00c8-41d7-8440-bb98d19476f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"
Sep 17 16:43:10.293682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.293661    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"]
Sep 17 16:43:10.294260 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.294243    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-stxzq"]
Sep 17 16:43:10.297030 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.297016    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.299691 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.299677    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 16:43:10.307753 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.307737    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 16:43:10.307753 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.307744    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-z7zbp\""
Sep 17 16:43:10.307867 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.307812    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 16:43:10.310546 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.310527    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cq9n8"]
Sep 17 16:43:10.313324 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.313309    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.315507 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.315483    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-stxzq"]
Sep 17 16:43:10.316120 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.316101    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wwxz2\""
Sep 17 16:43:10.316210 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.316177    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 16:43:10.316357 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.316339    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 16:43:10.316425 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.316365    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 16:43:10.390477 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.390448    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a691796a-00c8-41d7-8440-bb98d19476f5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-k4cjr\" (UID: \"a691796a-00c8-41d7-8440-bb98d19476f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"
Sep 17 16:43:10.390598 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.390509    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a62e64e0-3747-4257-ad9e-fb329dd8ac70-node-exporter-textfile\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.390648 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.390614    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62e64e0-3747-4257-ad9e-fb329dd8ac70-metrics-client-ca\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.390687 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.390644    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a62e64e0-3747-4257-ad9e-fb329dd8ac70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.390733 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.390714    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a62e64e0-3747-4257-ad9e-fb329dd8ac70-sys\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.390776 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.390761    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7pqr\" (UniqueName: \"kubernetes.io/projected/a62e64e0-3747-4257-ad9e-fb329dd8ac70-kube-api-access-b7pqr\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.390810 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.390792    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b9d6184f-c101-4444-bcdf-c935a87c813c-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.390840 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.390816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9d6184f-c101-4444-bcdf-c935a87c813c-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.390876 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.390843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zwk\" (UniqueName: \"kubernetes.io/projected/a691796a-00c8-41d7-8440-bb98d19476f5-kube-api-access-67zwk\") pod \"openshift-state-metrics-86c5bc79d6-k4cjr\" (UID: \"a691796a-00c8-41d7-8440-bb98d19476f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"
Sep 17 16:43:10.390945 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.390872    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9d6184f-c101-4444-bcdf-c935a87c813c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.390945 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.390889    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a62e64e0-3747-4257-ad9e-fb329dd8ac70-root\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.390945 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.390908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a691796a-00c8-41d7-8440-bb98d19476f5-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-k4cjr\" (UID: \"a691796a-00c8-41d7-8440-bb98d19476f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"
Sep 17 16:43:10.391095 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.390970    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a62e64e0-3747-4257-ad9e-fb329dd8ac70-node-exporter-tls\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.391095 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.391007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a62e64e0-3747-4257-ad9e-fb329dd8ac70-node-exporter-accelerators-collector-config\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.391095 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.391037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a62e64e0-3747-4257-ad9e-fb329dd8ac70-node-exporter-wtmp\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.391095 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.391060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b9d6184f-c101-4444-bcdf-c935a87c813c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.391301 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.391105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a691796a-00c8-41d7-8440-bb98d19476f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-k4cjr\" (UID: \"a691796a-00c8-41d7-8440-bb98d19476f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"
Sep 17 16:43:10.391301 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.391149    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9d6184f-c101-4444-bcdf-c935a87c813c-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.391301 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.391168    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xx89\" (UniqueName: \"kubernetes.io/projected/b9d6184f-c101-4444-bcdf-c935a87c813c-kube-api-access-5xx89\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.391301 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:43:10.391196    2565 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 17 16:43:10.391301 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:43:10.391247    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a691796a-00c8-41d7-8440-bb98d19476f5-openshift-state-metrics-tls podName:a691796a-00c8-41d7-8440-bb98d19476f5 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:10.891235196 +0000 UTC m=+56.457346900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a691796a-00c8-41d7-8440-bb98d19476f5-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-k4cjr" (UID: "a691796a-00c8-41d7-8440-bb98d19476f5") : secret "openshift-state-metrics-tls" not found
Sep 17 16:43:10.391667 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.391644    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a691796a-00c8-41d7-8440-bb98d19476f5-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-k4cjr\" (UID: \"a691796a-00c8-41d7-8440-bb98d19476f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"
Sep 17 16:43:10.394450 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.394428    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a691796a-00c8-41d7-8440-bb98d19476f5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-k4cjr\" (UID: \"a691796a-00c8-41d7-8440-bb98d19476f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"
Sep 17 16:43:10.411521 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.411498    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zwk\" (UniqueName: \"kubernetes.io/projected/a691796a-00c8-41d7-8440-bb98d19476f5-kube-api-access-67zwk\") pod \"openshift-state-metrics-86c5bc79d6-k4cjr\" (UID: \"a691796a-00c8-41d7-8440-bb98d19476f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"
Sep 17 16:43:10.413048 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.413019    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:10.491501 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.491474    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a62e64e0-3747-4257-ad9e-fb329dd8ac70-sys\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.491600 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.491510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7pqr\" (UniqueName: \"kubernetes.io/projected/a62e64e0-3747-4257-ad9e-fb329dd8ac70-kube-api-access-b7pqr\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.491600 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.491536    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b9d6184f-c101-4444-bcdf-c935a87c813c-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.491600 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.491562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9d6184f-c101-4444-bcdf-c935a87c813c-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.491600 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.491573    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a62e64e0-3747-4257-ad9e-fb329dd8ac70-sys\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.491600 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.491594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9d6184f-c101-4444-bcdf-c935a87c813c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.491846 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.491620    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a62e64e0-3747-4257-ad9e-fb329dd8ac70-root\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.491846 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.491654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a62e64e0-3747-4257-ad9e-fb329dd8ac70-node-exporter-tls\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.491846 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.491715    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a62e64e0-3747-4257-ad9e-fb329dd8ac70-root\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.491846 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.491789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a62e64e0-3747-4257-ad9e-fb329dd8ac70-node-exporter-accelerators-collector-config\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.491846 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.491830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a62e64e0-3747-4257-ad9e-fb329dd8ac70-node-exporter-wtmp\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.492138 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.491860    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b9d6184f-c101-4444-bcdf-c935a87c813c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.492138 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.491882    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b9d6184f-c101-4444-bcdf-c935a87c813c-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.492138 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.491908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9d6184f-c101-4444-bcdf-c935a87c813c-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.492138 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.491956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xx89\" (UniqueName: \"kubernetes.io/projected/b9d6184f-c101-4444-bcdf-c935a87c813c-kube-api-access-5xx89\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.492138 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.492002    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a62e64e0-3747-4257-ad9e-fb329dd8ac70-node-exporter-textfile\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.492138 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.492032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a62e64e0-3747-4257-ad9e-fb329dd8ac70-node-exporter-wtmp\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.492429 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.492276    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62e64e0-3747-4257-ad9e-fb329dd8ac70-metrics-client-ca\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.492429 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.492291    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a62e64e0-3747-4257-ad9e-fb329dd8ac70-node-exporter-textfile\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.492429 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.492311    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a62e64e0-3747-4257-ad9e-fb329dd8ac70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.492574 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.492434    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a62e64e0-3747-4257-ad9e-fb329dd8ac70-node-exporter-accelerators-collector-config\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.492664 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.492640    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9d6184f-c101-4444-bcdf-c935a87c813c-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.492841 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.492819    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62e64e0-3747-4257-ad9e-fb329dd8ac70-metrics-client-ca\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.493036 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.493016    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b9d6184f-c101-4444-bcdf-c935a87c813c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.494012 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.493991    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a62e64e0-3747-4257-ad9e-fb329dd8ac70-node-exporter-tls\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.494370 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.494353    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9d6184f-c101-4444-bcdf-c935a87c813c-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.494463 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.494443    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a62e64e0-3747-4257-ad9e-fb329dd8ac70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.494500 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.494451    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9d6184f-c101-4444-bcdf-c935a87c813c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.502563 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.502538    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7pqr\" (UniqueName: \"kubernetes.io/projected/a62e64e0-3747-4257-ad9e-fb329dd8ac70-kube-api-access-b7pqr\") pod \"node-exporter-cq9n8\" (UID: \"a62e64e0-3747-4257-ad9e-fb329dd8ac70\") " pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.503564 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.503540    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xx89\" (UniqueName: \"kubernetes.io/projected/b9d6184f-c101-4444-bcdf-c935a87c813c-kube-api-access-5xx89\") pod \"kube-state-metrics-84c885658c-stxzq\" (UID: \"b9d6184f-c101-4444-bcdf-c935a87c813c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.503779 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.503762    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:10.605029 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.604980    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq"
Sep 17 16:43:10.622361 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.622334    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cq9n8"
Sep 17 16:43:10.628822 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.628794    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:10.631976 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:43:10.631949    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62e64e0_3747_4257_ad9e_fb329dd8ac70.slice/crio-6d43e5ac2e77bfd595b9bf1ed5de5c8a2c9ab7cad0ac41d8721e8157cd083e08 WatchSource:0}: Error finding container 6d43e5ac2e77bfd595b9bf1ed5de5c8a2c9ab7cad0ac41d8721e8157cd083e08: Status 404 returned error can't find the container with id 6d43e5ac2e77bfd595b9bf1ed5de5c8a2c9ab7cad0ac41d8721e8157cd083e08
Sep 17 16:43:10.728724 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.728696    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:10.788985 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.788958    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:10.879589 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.879565    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-stxzq"]
Sep 17 16:43:10.882074 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:43:10.882048    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d6184f_c101_4444_bcdf_c935a87c813c.slice/crio-2d807f76f78f63c521da0210974f1d0d8f548eb4cc61f8fb28db63ddec1244bf WatchSource:0}: Error finding container 2d807f76f78f63c521da0210974f1d0d8f548eb4cc61f8fb28db63ddec1244bf: Status 404 returned error can't find the container with id 2d807f76f78f63c521da0210974f1d0d8f548eb4cc61f8fb28db63ddec1244bf
Sep 17 16:43:10.895282 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.895255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a691796a-00c8-41d7-8440-bb98d19476f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-k4cjr\" (UID: \"a691796a-00c8-41d7-8440-bb98d19476f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"
Sep 17 16:43:10.897382 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:10.897360    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a691796a-00c8-41d7-8440-bb98d19476f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-k4cjr\" (UID: \"a691796a-00c8-41d7-8440-bb98d19476f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"
Sep 17 16:43:11.027803 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:11.027776    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cq9n8" event={"ID":"a62e64e0-3747-4257-ad9e-fb329dd8ac70","Type":"ContainerStarted","Data":"6d43e5ac2e77bfd595b9bf1ed5de5c8a2c9ab7cad0ac41d8721e8157cd083e08"}
Sep 17 16:43:11.028984 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:11.028960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq" event={"ID":"b9d6184f-c101-4444-bcdf-c935a87c813c","Type":"ContainerStarted","Data":"2d807f76f78f63c521da0210974f1d0d8f548eb4cc61f8fb28db63ddec1244bf"}
Sep 17 16:43:11.177174 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:11.177118    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"
Sep 17 16:43:11.270904 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:11.270876    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:11.355962 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:11.355931    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:11.458402 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:11.458334    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr"]
Sep 17 16:43:11.462412 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:11.462355    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:11.649746 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:43:11.649710    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda691796a_00c8_41d7_8440_bb98d19476f5.slice/crio-ec1bb244e33037f36da8c297ef68dc4b807e369ca00ac5d04d7b7dc03e017fe0 WatchSource:0}: Error finding container ec1bb244e33037f36da8c297ef68dc4b807e369ca00ac5d04d7b7dc03e017fe0: Status 404 returned error can't find the container with id ec1bb244e33037f36da8c297ef68dc4b807e369ca00ac5d04d7b7dc03e017fe0
Sep 17 16:43:11.742396 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:11.742322    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:12.032790 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.032762    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr" event={"ID":"a691796a-00c8-41d7-8440-bb98d19476f5","Type":"ContainerStarted","Data":"ec1bb244e33037f36da8c297ef68dc4b807e369ca00ac5d04d7b7dc03e017fe0"}
Sep 17 16:43:12.287982 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.285160    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-54576c98d9-dm9pd"]
Sep 17 16:43:12.292193 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.291951    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.294848 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.294819    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 16:43:12.294848 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.294832    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 16:43:12.295051 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.294992    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 16:43:12.295138 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.295118    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 16:43:12.295360 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.295344    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-p8x4q\""
Sep 17 16:43:12.295875 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.295858    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6m243doanojtk\""
Sep 17 16:43:12.295950 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.295904    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 16:43:12.299950 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.299902    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54576c98d9-dm9pd"]
Sep 17 16:43:12.306810 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.306784    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-thanos-querier-tls\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.306883 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.306849    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.306883 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.306876    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p65x8\" (UniqueName: \"kubernetes.io/projected/d6bee9be-246a-4c3f-8f63-656d809eb7d9-kube-api-access-p65x8\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.306974 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.306925    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-grpc-tls\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.306974 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.306965    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6bee9be-246a-4c3f-8f63-656d809eb7d9-metrics-client-ca\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.307037 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.307008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.307037 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.307027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.307095 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.307059    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.407831 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.407808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.407961 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.407838    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p65x8\" (UniqueName: \"kubernetes.io/projected/d6bee9be-246a-4c3f-8f63-656d809eb7d9-kube-api-access-p65x8\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.407961 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.407860    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-grpc-tls\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.407961 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.407876    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6bee9be-246a-4c3f-8f63-656d809eb7d9-metrics-client-ca\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.408168 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.408030    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.408168 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.408080    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.408168 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.408141    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.408311 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.408182    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-thanos-querier-tls\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.408861 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.408835    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6bee9be-246a-4c3f-8f63-656d809eb7d9-metrics-client-ca\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.410823 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.410801    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.410889 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.410838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-grpc-tls\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.411297 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.411276    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.411376 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.411276    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.411464 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.411445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.411500 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.411455    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6bee9be-246a-4c3f-8f63-656d809eb7d9-secret-thanos-querier-tls\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.416282 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.416263    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p65x8\" (UniqueName: \"kubernetes.io/projected/d6bee9be-246a-4c3f-8f63-656d809eb7d9-kube-api-access-p65x8\") pod \"thanos-querier-54576c98d9-dm9pd\" (UID: \"d6bee9be-246a-4c3f-8f63-656d809eb7d9\") " pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.600732 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.600661    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:12.878072 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:12.878049    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54576c98d9-dm9pd"]
Sep 17 16:43:12.879243 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:43:12.879223    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6bee9be_246a_4c3f_8f63_656d809eb7d9.slice/crio-428e3de7038710f1dfe366d3210fd71ccc393697cf5171e6f2eb04e3a23cc3af WatchSource:0}: Error finding container 428e3de7038710f1dfe366d3210fd71ccc393697cf5171e6f2eb04e3a23cc3af: Status 404 returned error can't find the container with id 428e3de7038710f1dfe366d3210fd71ccc393697cf5171e6f2eb04e3a23cc3af
Sep 17 16:43:13.038840 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:13.038804    2565 generic.go:356] "Generic (PLEG): container finished" podID="a62e64e0-3747-4257-ad9e-fb329dd8ac70" containerID="3452155db81a9eec9360fba8c9e748dd7649df0a04e3b45b16ec746eaa850a01" exitCode=0
Sep 17 16:43:13.038998 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:13.038888    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cq9n8" event={"ID":"a62e64e0-3747-4257-ad9e-fb329dd8ac70","Type":"ContainerDied","Data":"3452155db81a9eec9360fba8c9e748dd7649df0a04e3b45b16ec746eaa850a01"}
Sep 17 16:43:13.040342 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:13.040315    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd" event={"ID":"d6bee9be-246a-4c3f-8f63-656d809eb7d9","Type":"ContainerStarted","Data":"428e3de7038710f1dfe366d3210fd71ccc393697cf5171e6f2eb04e3a23cc3af"}
Sep 17 16:43:13.042502 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:13.042473    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr" event={"ID":"a691796a-00c8-41d7-8440-bb98d19476f5","Type":"ContainerStarted","Data":"3e35111d7902542d4e17ebeb8a7c7d9b3a591fd3a9dae4354c8e2375ca09c011"}
Sep 17 16:43:13.042656 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:13.042529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr" event={"ID":"a691796a-00c8-41d7-8440-bb98d19476f5","Type":"ContainerStarted","Data":"01c0f413ac2f2da3828d63bef4bc0406f8ed3e575099f4a77affa95124b53230"}
Sep 17 16:43:13.044948 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:13.044899    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq" event={"ID":"b9d6184f-c101-4444-bcdf-c935a87c813c","Type":"ContainerStarted","Data":"3bcc371f857bae70846141fdbe9aa0c1f827dd6d4174fd9ac448886e8b31379d"}
Sep 17 16:43:13.045042 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:13.044966    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq" event={"ID":"b9d6184f-c101-4444-bcdf-c935a87c813c","Type":"ContainerStarted","Data":"b10d5868be6be394e245576fb67535d05af74310abf2811e3c2a1bf5907a9f16"}
Sep 17 16:43:13.045042 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:13.044994    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq" event={"ID":"b9d6184f-c101-4444-bcdf-c935a87c813c","Type":"ContainerStarted","Data":"a3d20048a475cecf55581a4a09ab5bd4a7f3791d14f8037932c74dca9faf836a"}
Sep 17 16:43:13.073898 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:13.073844    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-stxzq" podStartSLOduration=1.958748961 podStartE2EDuration="3.07382815s" totalImagesPullingTime="1.115079189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:13.073058151 +0000 UTC m=+58.639169887" watchObservedRunningTime="2026-09-17 16:43:13.07382815 +0000 UTC m=+58.639939875"
Sep 17 16:43:13.234366 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:13.234325    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:13.347123 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:13.347092    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:13.521370 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:13.521337    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:13.947352 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:13.947326    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q7v29"
Sep 17 16:43:13.954255 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:13.954221    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:14.052768 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.052739    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cq9n8" event={"ID":"a62e64e0-3747-4257-ad9e-fb329dd8ac70","Type":"ContainerStarted","Data":"65c4a6a7c6c00794e640e7b5ac506c558b8537f6f5227ae347dc5b4187d2b653"}
Sep 17 16:43:14.052930 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.052820    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cq9n8" event={"ID":"a62e64e0-3747-4257-ad9e-fb329dd8ac70","Type":"ContainerStarted","Data":"47f008dfb20ef5c7e44168a359d0479f26a59d3104a03fea20ac9df7f3e0f86a"}
Sep 17 16:43:14.054696 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.054668    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr" event={"ID":"a691796a-00c8-41d7-8440-bb98d19476f5","Type":"ContainerStarted","Data":"2120c453be4cb618ded33c57c57344e89ca58eb3a3a7993c5eb87d1181e2a65b"}
Sep 17 16:43:14.072784 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.072734    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cq9n8" podStartSLOduration=2.6249761019999998 podStartE2EDuration="4.072719018s" totalImagesPullingTime="1.364235672s" totalInitContainerRuntime="83.507244ms" isStatefulPod=true podCreationTimestamp="2026-09-17 16:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:14.07123161 +0000 UTC m=+59.637343333" watchObservedRunningTime="2026-09-17 16:43:14.072719018 +0000 UTC m=+59.638830740"
Sep 17 16:43:14.076043 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.076017    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:14.087886 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.087845    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k4cjr" podStartSLOduration=2.835541172 podStartE2EDuration="4.087834898s" totalImagesPullingTime="1.252293726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:14.087011288 +0000 UTC m=+59.653123009" watchObservedRunningTime="2026-09-17 16:43:14.087834898 +0000 UTC m=+59.653946619"
Sep 17 16:43:14.153682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.153661    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:14.562440 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.562408    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:14.610790 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.610759    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"]
Sep 17 16:43:14.628702 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.628680    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"]
Sep 17 16:43:14.628802 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.628761    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.631823 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.631803    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 16:43:14.631942 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.631856    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-qk7tejkvt8k6\""
Sep 17 16:43:14.633022 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.632997    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 16:43:14.633022 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.633013    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 16:43:14.633171 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.633116    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cpbmt\""
Sep 17 16:43:14.633237 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.633222    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 16:43:14.650164 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.650143    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:14.724617 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.724594    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/da4f2ef7-b89c-4a07-ad96-188207e336bf-audit-log\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.724708 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.724626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/da4f2ef7-b89c-4a07-ad96-188207e336bf-secret-metrics-server-client-certs\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.724708 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.724669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6pq4\" (UniqueName: \"kubernetes.io/projected/da4f2ef7-b89c-4a07-ad96-188207e336bf-kube-api-access-t6pq4\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.724821 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.724769    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/da4f2ef7-b89c-4a07-ad96-188207e336bf-client-ca-bundle\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.724821 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.724813    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/da4f2ef7-b89c-4a07-ad96-188207e336bf-metrics-server-audit-profiles\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.724956 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.724868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/da4f2ef7-b89c-4a07-ad96-188207e336bf-secret-metrics-server-tls\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.724956 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.724897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da4f2ef7-b89c-4a07-ad96-188207e336bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.825334 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.825290    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/da4f2ef7-b89c-4a07-ad96-188207e336bf-client-ca-bundle\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.825334 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.825316    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/da4f2ef7-b89c-4a07-ad96-188207e336bf-metrics-server-audit-profiles\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.825446 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.825353    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/da4f2ef7-b89c-4a07-ad96-188207e336bf-secret-metrics-server-tls\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.825446 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.825381    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da4f2ef7-b89c-4a07-ad96-188207e336bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.825446 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.825424    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/da4f2ef7-b89c-4a07-ad96-188207e336bf-audit-log\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.825578 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.825459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/da4f2ef7-b89c-4a07-ad96-188207e336bf-secret-metrics-server-client-certs\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.825578 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.825501    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6pq4\" (UniqueName: \"kubernetes.io/projected/da4f2ef7-b89c-4a07-ad96-188207e336bf-kube-api-access-t6pq4\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.825819 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.825789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/da4f2ef7-b89c-4a07-ad96-188207e336bf-audit-log\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.826183 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.826160    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da4f2ef7-b89c-4a07-ad96-188207e336bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.826288 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.826271    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/da4f2ef7-b89c-4a07-ad96-188207e336bf-metrics-server-audit-profiles\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.827674 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.827655    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/da4f2ef7-b89c-4a07-ad96-188207e336bf-client-ca-bundle\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.827880 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.827860    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/da4f2ef7-b89c-4a07-ad96-188207e336bf-secret-metrics-server-tls\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.827945 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.827891    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/da4f2ef7-b89c-4a07-ad96-188207e336bf-secret-metrics-server-client-certs\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.833668 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.833650    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6pq4\" (UniqueName: \"kubernetes.io/projected/da4f2ef7-b89c-4a07-ad96-188207e336bf-kube-api-access-t6pq4\") pod \"metrics-server-6fdd44fc8b-kzwf5\" (UID: \"da4f2ef7-b89c-4a07-ad96-188207e336bf\") " pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:14.937393 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:14.937370    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:15.032925 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.032890    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:15.045211 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.045187    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-tbsbh"]
Sep 17 16:43:15.070772 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.070753    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-tbsbh"]
Sep 17 16:43:15.070859 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.070835    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-tbsbh"
Sep 17 16:43:15.073246 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.073227    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 16:43:15.073364 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.073350    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xfqzb\""
Sep 17 16:43:15.091473 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.091455    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:15.129305 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.129279    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmt5s\" (UniqueName: \"kubernetes.io/projected/95994e0a-0887-4830-8d71-2a2aac599932-kube-api-access-xmt5s\") pod \"monitoring-plugin-5984459ff8-tbsbh\" (UID: \"95994e0a-0887-4830-8d71-2a2aac599932\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-tbsbh"
Sep 17 16:43:15.129432 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.129368    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/95994e0a-0887-4830-8d71-2a2aac599932-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-tbsbh\" (UID: \"95994e0a-0887-4830-8d71-2a2aac599932\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-tbsbh"
Sep 17 16:43:15.146995 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.146967    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:15.222191 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.222167    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"]
Sep 17 16:43:15.225015 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:43:15.224989    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4f2ef7_b89c_4a07_ad96_188207e336bf.slice/crio-4e993fdb7d6bc45d1700424e99a1aa61555d8dfa2f9df3eb7532ffa9aa48c653 WatchSource:0}: Error finding container 4e993fdb7d6bc45d1700424e99a1aa61555d8dfa2f9df3eb7532ffa9aa48c653: Status 404 returned error can't find the container with id 4e993fdb7d6bc45d1700424e99a1aa61555d8dfa2f9df3eb7532ffa9aa48c653
Sep 17 16:43:15.230171 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.230151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmt5s\" (UniqueName: \"kubernetes.io/projected/95994e0a-0887-4830-8d71-2a2aac599932-kube-api-access-xmt5s\") pod \"monitoring-plugin-5984459ff8-tbsbh\" (UID: \"95994e0a-0887-4830-8d71-2a2aac599932\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-tbsbh"
Sep 17 16:43:15.230254 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.230201    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/95994e0a-0887-4830-8d71-2a2aac599932-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-tbsbh\" (UID: \"95994e0a-0887-4830-8d71-2a2aac599932\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-tbsbh"
Sep 17 16:43:15.232863 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.232842    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/95994e0a-0887-4830-8d71-2a2aac599932-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-tbsbh\" (UID: \"95994e0a-0887-4830-8d71-2a2aac599932\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-tbsbh"
Sep 17 16:43:15.237951 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.237899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmt5s\" (UniqueName: \"kubernetes.io/projected/95994e0a-0887-4830-8d71-2a2aac599932-kube-api-access-xmt5s\") pod \"monitoring-plugin-5984459ff8-tbsbh\" (UID: \"95994e0a-0887-4830-8d71-2a2aac599932\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-tbsbh"
Sep 17 16:43:15.379013 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.378989    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-tbsbh"
Sep 17 16:43:15.660168 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.660147    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-tbsbh"]
Sep 17 16:43:15.661976 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:43:15.661953    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95994e0a_0887_4830_8d71_2a2aac599932.slice/crio-a4a09dbdd63b2632345370a44698a09c76166a29bed65248bc49f0c3872f64d0 WatchSource:0}: Error finding container a4a09dbdd63b2632345370a44698a09c76166a29bed65248bc49f0c3872f64d0: Status 404 returned error can't find the container with id a4a09dbdd63b2632345370a44698a09c76166a29bed65248bc49f0c3872f64d0
Sep 17 16:43:15.799588 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.799560    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:15.901174 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:15.901141    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:16.065254 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.065173    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd" event={"ID":"d6bee9be-246a-4c3f-8f63-656d809eb7d9","Type":"ContainerStarted","Data":"b39d91ded88ec25bed25637c19eb556bcbd0ab4b130b28178634726837b0311d"}
Sep 17 16:43:16.065667 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.065266    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd" event={"ID":"d6bee9be-246a-4c3f-8f63-656d809eb7d9","Type":"ContainerStarted","Data":"aa8a8f76b6eb3df37574aa7b96fc62bf581dd8b583973c86e9182c8d38f547b2"}
Sep 17 16:43:16.065667 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.065308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd" event={"ID":"d6bee9be-246a-4c3f-8f63-656d809eb7d9","Type":"ContainerStarted","Data":"0aa0697ab12fdfc4d6fd1c172eaa85fd53ed860d2298c218429106f9b747df0e"}
Sep 17 16:43:16.066789 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.066734    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-tbsbh" event={"ID":"95994e0a-0887-4830-8d71-2a2aac599932","Type":"ContainerStarted","Data":"a4a09dbdd63b2632345370a44698a09c76166a29bed65248bc49f0c3872f64d0"}
Sep 17 16:43:16.068234 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.068205    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5" event={"ID":"da4f2ef7-b89c-4a07-ad96-188207e336bf","Type":"ContainerStarted","Data":"4e993fdb7d6bc45d1700424e99a1aa61555d8dfa2f9df3eb7532ffa9aa48c653"}
Sep 17 16:43:16.494105 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.494083    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:16.499762 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.499729    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 16:43:16.502318 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.502299    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.504703 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.504681    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 16:43:16.505187 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.505015    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 16:43:16.505187 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.505037    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 16:43:16.505187 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.505037    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 16:43:16.505187 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.505081    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 16:43:16.505450 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.505325    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 16:43:16.505450 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.505387    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 16:43:16.505567 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.505548    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 16:43:16.505669 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.505615    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 16:43:16.505669 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.505626    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 16:43:16.505748 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.505731    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4bocskh2vfiqn\""
Sep 17 16:43:16.506501 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.506257    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qvc78\""
Sep 17 16:43:16.506501 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.506275    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 16:43:16.506501 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.506316    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 16:43:16.506501 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.506423    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 16:43:16.508863 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.508846    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 16:43:16.526468 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.526318    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 16:43:16.542336 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542288    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-web-config\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.542438 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-config-out\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.542438 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542410    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.542567 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542437    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.542567 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542464    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.542567 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542487    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.542567 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.542567 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542556    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.542800 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542585    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.542800 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.542800 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542641    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.542800 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-config\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.542800 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542695    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.542800 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542736    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.542800 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542768    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.542800 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542798    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.543181 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.543181 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.543181 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.543181 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.542932    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4xvw\" (UniqueName: \"kubernetes.io/projected/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-kube-api-access-d4xvw\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.643792 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.643767    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.643873 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.643809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.643873 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.643837    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.643873 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.643862    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4xvw\" (UniqueName: \"kubernetes.io/projected/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-kube-api-access-d4xvw\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.644062 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.643889    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-web-config\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.644062 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.643960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-config-out\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.644062 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.643994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.644062 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.644017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.644062 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.644049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.644437 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.644070    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.644437 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.644091    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.644437 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.644139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.644437 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.644166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.644437 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.644201    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.644437 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.644224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.644437 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.644253    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-config\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.644437 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.644282    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.644437 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.644323    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.644855 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.644436    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.644855 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.644475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.645767 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.645736    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.646043 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.646020    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.646864 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.646842    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-web-config\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.647176 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.647151    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.647389 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.647361    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.648430 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.648401    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.648780 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.648755    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.649339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.649316    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.649709 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.649687    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.652547 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.652525    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.654258 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.654208    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-config\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.654622 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.654599    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-config-out\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.655456 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.655412    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.656076 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.656025    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.656735 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.656698    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.656822 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.656744    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.656822 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.656765    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.657751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.657260    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.657751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.657472    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.659806 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.659785    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4xvw\" (UniqueName: \"kubernetes.io/projected/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-kube-api-access-d4xvw\") pod \"prometheus-k8s-0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:16.816392 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.816320    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qvc78\""
Sep 17 16:43:16.824645 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:16.824618    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:17.077681 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:17.077654    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd" event={"ID":"d6bee9be-246a-4c3f-8f63-656d809eb7d9","Type":"ContainerStarted","Data":"07b0e8c4bf9567474185449c52078b07d047b93082c7b172309286e69a601074"}
Sep 17 16:43:17.080568 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:17.077720    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd" event={"ID":"d6bee9be-246a-4c3f-8f63-656d809eb7d9","Type":"ContainerStarted","Data":"db205685a1c618d536065d452640a3717f851f745042509ee8752a9b365f05a8"}
Sep 17 16:43:17.080568 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:17.077747    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd" event={"ID":"d6bee9be-246a-4c3f-8f63-656d809eb7d9","Type":"ContainerStarted","Data":"564b1e1653b716077ddfed62ce733da3c20e2526d55c1116ed4bed2c36f7a2ad"}
Sep 17 16:43:17.080568 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:17.077940    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:17.080568 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:17.079983    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5" event={"ID":"da4f2ef7-b89c-4a07-ad96-188207e336bf","Type":"ContainerStarted","Data":"15b6c56ac83b97b78c1c4d108414873426c6fc91c86468f0b67c8355b9953a96"}
Sep 17 16:43:17.100474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:17.100426    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd" podStartSLOduration=1.839036503 podStartE2EDuration="5.100406815s" totalImagesPullingTime="3.261370312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:17.099546603 +0000 UTC m=+62.665658315" watchObservedRunningTime="2026-09-17 16:43:17.100406815 +0000 UTC m=+62.666518538"
Sep 17 16:43:17.117476 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:17.117434    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5" podStartSLOduration=1.829740348 podStartE2EDuration="3.117419445s" totalImagesPullingTime="1.287679097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:17.116301134 +0000 UTC m=+62.682412857" watchObservedRunningTime="2026-09-17 16:43:17.117419445 +0000 UTC m=+62.683531168"
Sep 17 16:43:17.307302 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:17.302903    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 16:43:17.743887 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:17.743856    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:17.825522 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:17.825494    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:18.085353 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:18.085272    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-tbsbh" event={"ID":"95994e0a-0887-4830-8d71-2a2aac599932","Type":"ContainerStarted","Data":"7e8b841670fba263a1ca0f8dab076fec683df6a3223ec6a6820714b004814af0"}
Sep 17 16:43:18.085768 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:18.085435    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-tbsbh"
Sep 17 16:43:18.086971 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:18.086907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0","Type":"ContainerStarted","Data":"5bf6a06dd35fd4edc5016bfabe3800e40e851b35375667c3b3c595783915a44c"}
Sep 17 16:43:18.091483 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:18.091461    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-tbsbh"
Sep 17 16:43:18.101180 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:18.101136    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-tbsbh" podStartSLOduration=1.713797528 podStartE2EDuration="3.101119345s" totalImagesPullingTime="1.387321817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:18.099947229 +0000 UTC m=+63.666058948" watchObservedRunningTime="2026-09-17 16:43:18.101119345 +0000 UTC m=+63.667231069"
Sep 17 16:43:19.091925 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:19.091889    2565 generic.go:356] "Generic (PLEG): container finished" podID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerID="c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea" exitCode=0
Sep 17 16:43:19.092273 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:19.091937    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0","Type":"ContainerDied","Data":"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea"}
Sep 17 16:43:19.789382 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:19.789345    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:19.887099 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:19.887067    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:20.031711 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:20.031679    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-28z7f"
Sep 17 16:43:20.106410 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:20.106332    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:20.207021 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:20.206986    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:20.647014 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:20.646981    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:20.746475 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:20.746444    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:20.847573 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:20.847539    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:22.497238 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.497210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs\") pod \"network-metrics-daemon-wld7n\" (UID: \"716feac9-5ed2-4b28-9ede-a210a4871e3d\") " pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:43:22.501734 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.501708    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 16:43:22.513748 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.513715    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716feac9-5ed2-4b28-9ede-a210a4871e3d-metrics-certs\") pod \"network-metrics-daemon-wld7n\" (UID: \"716feac9-5ed2-4b28-9ede-a210a4871e3d\") " pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:43:22.598478 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.598454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret\") pod \"global-pull-secret-syncer-5llzd\" (UID: \"70f3faf3-b150-468f-abb7-40f2d221164e\") " pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:43:22.598580 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.598504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tqx\" (UniqueName: \"kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx\") pod \"network-check-target-zl97d\" (UID: \"c0a61d36-0e50-4539-ac3f-d5b6c9c3e287\") " pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:43:22.600934 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.600890    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 16:43:22.601046 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.600891    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 16:43:22.611505 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.611488    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 16:43:22.611505 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.611499    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70f3faf3-b150-468f-abb7-40f2d221164e-original-pull-secret\") pod \"global-pull-secret-syncer-5llzd\" (UID: \"70f3faf3-b150-468f-abb7-40f2d221164e\") " pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:43:22.622011 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.621990    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4tqx\" (UniqueName: \"kubernetes.io/projected/c0a61d36-0e50-4539-ac3f-d5b6c9c3e287-kube-api-access-p4tqx\") pod \"network-check-target-zl97d\" (UID: \"c0a61d36-0e50-4539-ac3f-d5b6c9c3e287\") " pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:43:22.734298 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.734275    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ls2kp\""
Sep 17 16:43:22.738908 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.738892    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kh76t\""
Sep 17 16:43:22.740955 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.740934    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5llzd"
Sep 17 16:43:22.741642 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.741623    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wld7n"
Sep 17 16:43:22.744751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.744726    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:22.746718 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.746700    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:43:22.843965 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:22.843904    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:23.024851 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:23.024639    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5llzd"]
Sep 17 16:43:23.028052 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:43:23.027928    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f3faf3_b150_468f_abb7_40f2d221164e.slice/crio-152f7f0c9112e863ac166800348694672f3804167ff24bd4584d99761a3c936c WatchSource:0}: Error finding container 152f7f0c9112e863ac166800348694672f3804167ff24bd4584d99761a3c936c: Status 404 returned error can't find the container with id 152f7f0c9112e863ac166800348694672f3804167ff24bd4584d99761a3c936c
Sep 17 16:43:23.028109 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:23.028047    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wld7n"]
Sep 17 16:43:23.030365 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:43:23.030329    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716feac9_5ed2_4b28_9ede_a210a4871e3d.slice/crio-18fe3ef397b6988110e02365f8d76ba4672898932e921b94a26d55e20cd616c0 WatchSource:0}: Error finding container 18fe3ef397b6988110e02365f8d76ba4672898932e921b94a26d55e20cd616c0: Status 404 returned error can't find the container with id 18fe3ef397b6988110e02365f8d76ba4672898932e921b94a26d55e20cd616c0
Sep 17 16:43:23.037071 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:23.037048    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zl97d"]
Sep 17 16:43:23.040012 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:43:23.039987    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a61d36_0e50_4539_ac3f_d5b6c9c3e287.slice/crio-7e154bd67a42803dd864c1070ba26dc69e4874e09f382a7474097ccb534220b4 WatchSource:0}: Error finding container 7e154bd67a42803dd864c1070ba26dc69e4874e09f382a7474097ccb534220b4: Status 404 returned error can't find the container with id 7e154bd67a42803dd864c1070ba26dc69e4874e09f382a7474097ccb534220b4
Sep 17 16:43:23.092823 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:23.092801    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-54576c98d9-dm9pd"
Sep 17 16:43:23.117795 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:23.117769    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0","Type":"ContainerStarted","Data":"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb"}
Sep 17 16:43:23.117908 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:23.117833    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0","Type":"ContainerStarted","Data":"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed"}
Sep 17 16:43:23.117908 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:23.117871    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0","Type":"ContainerStarted","Data":"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd"}
Sep 17 16:43:23.118043 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:23.117940    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0","Type":"ContainerStarted","Data":"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3"}
Sep 17 16:43:23.118043 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:23.117970    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0","Type":"ContainerStarted","Data":"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192"}
Sep 17 16:43:23.118043 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:23.118008    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0","Type":"ContainerStarted","Data":"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099"}
Sep 17 16:43:23.119001 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:23.118952    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zl97d" event={"ID":"c0a61d36-0e50-4539-ac3f-d5b6c9c3e287","Type":"ContainerStarted","Data":"7e154bd67a42803dd864c1070ba26dc69e4874e09f382a7474097ccb534220b4"}
Sep 17 16:43:23.121106 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:23.121083    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5llzd" event={"ID":"70f3faf3-b150-468f-abb7-40f2d221164e","Type":"ContainerStarted","Data":"152f7f0c9112e863ac166800348694672f3804167ff24bd4584d99761a3c936c"}
Sep 17 16:43:23.122186 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:23.122168    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wld7n" event={"ID":"716feac9-5ed2-4b28-9ede-a210a4871e3d","Type":"ContainerStarted","Data":"18fe3ef397b6988110e02365f8d76ba4672898932e921b94a26d55e20cd616c0"}
Sep 17 16:43:23.155059 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:23.155018    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.938401392 podStartE2EDuration="7.155004401s" totalImagesPullingTime="4.198550067s" totalInitContainerRuntime="18.052942ms" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:23.153336706 +0000 UTC m=+68.719448464" watchObservedRunningTime="2026-09-17 16:43:23.155004401 +0000 UTC m=+68.721116157"
Sep 17 16:43:24.814345 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:24.814302    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:24.915565 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:24.915538    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:25.135984 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:25.135950    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wld7n" event={"ID":"716feac9-5ed2-4b28-9ede-a210a4871e3d","Type":"ContainerStarted","Data":"32af4c58dc654a5b05af07aa78c8e4c56c48a57f42c5f00f5d343b3e909896fa"}
Sep 17 16:43:25.136152 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:25.136018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wld7n" event={"ID":"716feac9-5ed2-4b28-9ede-a210a4871e3d","Type":"ContainerStarted","Data":"6ccaa1f67a52031a1436bc849ff4fedb190266c6ef9e4992748baf3c2d3c4c28"}
Sep 17 16:43:25.158590 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:25.158536    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wld7n" podStartSLOduration=68.055138362 podStartE2EDuration="1m9.158517812s" totalImagesPullingTime="1.10337945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:25.157210716 +0000 UTC m=+70.723322475" watchObservedRunningTime="2026-09-17 16:43:25.158517812 +0000 UTC m=+70.724629535"
Sep 17 16:43:25.259034 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:25.259006    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:25.355257 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:25.355230    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:25.455367 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:25.455289    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:25.670718 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:25.670684    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:25.726030 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:25.725963    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:26.806678 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:26.806645    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:26.825754 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:26.825722    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:43:26.881000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:26.880907    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:27.898706 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:27.898673    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:28.151970 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:28.151859    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zl97d" event={"ID":"c0a61d36-0e50-4539-ac3f-d5b6c9c3e287","Type":"ContainerStarted","Data":"fbf60ffb3a61ec4631befd50543697261f29f1dddccdfd7bd3a2c62286ce616b"}
Sep 17 16:43:28.152132 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:28.151975    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:43:28.153558 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:28.153537    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5llzd" event={"ID":"70f3faf3-b150-468f-abb7-40f2d221164e","Type":"ContainerStarted","Data":"fa3afc620c5069deb87c3a4dde5f453ae30906d440eb99622c1798187bc63881"}
Sep 17 16:43:28.168343 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:28.168303    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zl97d" podStartSLOduration=67.300333628 podStartE2EDuration="1m12.168292113s" totalImagesPullingTime="4.867958485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:28.167519531 +0000 UTC m=+73.733631254" watchObservedRunningTime="2026-09-17 16:43:28.168292113 +0000 UTC m=+73.734403834"
Sep 17 16:43:28.182942 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:28.182875    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5llzd" podStartSLOduration=67.30438618 podStartE2EDuration="1m12.182859464s" totalImagesPullingTime="4.878473284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:28.182707721 +0000 UTC m=+73.748819445" watchObservedRunningTime="2026-09-17 16:43:28.182859464 +0000 UTC m=+73.748971187"
Sep 17 16:43:33.301560 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:33.301523    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:33.363730 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:33.363700    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:33.835326 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:33.835299    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:33.935391 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:33.935354    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:34.923268 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:34.923228    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:34.938155 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:34.938128    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:34.938303 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:34.938189    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:40.666036 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:40.666000    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:40.762473 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:40.762441    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:54.096034 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:54.096003    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:54.157352 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:54.157324    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:43:54.942811 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:54.942782    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:54.946516 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:54.946491    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6fdd44fc8b-kzwf5"
Sep 17 16:43:59.160813 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:43:59.160784    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zl97d"
Sep 17 16:44:07.689326 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:07.689294    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:09.580535 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:09.580493    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:09.644670 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:09.644636    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:16.825302 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:16.825269    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:16.843715 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:16.843693    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:17.346850 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:17.346823    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:29.564071 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:29.564033    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:30.141635 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:30.141574    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:34.954263 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:34.954233    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 16:44:34.954752 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:34.954518    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="prometheus" containerID="cri-o://1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099" gracePeriod=600
Sep 17 16:44:34.954752 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:34.954537    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="thanos-sidecar" containerID="cri-o://8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3" gracePeriod=600
Sep 17 16:44:34.954752 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:34.954558    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="kube-rbac-proxy" containerID="cri-o://1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed" gracePeriod=600
Sep 17 16:44:34.954752 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:34.954578    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="config-reloader" containerID="cri-o://102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192" gracePeriod=600
Sep 17 16:44:34.954752 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:34.954584    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="kube-rbac-proxy-web" containerID="cri-o://8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd" gracePeriod=600
Sep 17 16:44:34.954752 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:34.954590    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb" gracePeriod=600
Sep 17 16:44:35.181632 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.181607    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.221972 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.221892    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-2\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.221972 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.221954    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-0\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222154 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.221987    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4xvw\" (UniqueName: \"kubernetes.io/projected/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-kube-api-access-d4xvw\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222154 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222031    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-config\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222154 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222063    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-db\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222154 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222099    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-metrics-client-ca\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222154 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222124    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-tls-assets\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222154 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222148    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-1\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222462 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222184    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-kubelet-serving-ca-bundle\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222462 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222231    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-serving-certs-ca-bundle\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222462 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222260    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-thanos-prometheus-http-client-file\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222462 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222298    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-trusted-ca-bundle\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222462 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222322    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-tls\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222462 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222355    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-grpc-tls\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222462 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222223    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:35.222462 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222383    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-kube-rbac-proxy\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222462 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222413    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222462 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222440    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-metrics-client-certs\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222989 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222470    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-web-config\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222989 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222495    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222989 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222547    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-config-out\") pod \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\" (UID: \"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0\") "
Sep 17 16:44:35.222989 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222688    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:35.222989 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222770    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.222989 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.222787    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.223214 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.223109    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:35.223415 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.223393    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:35.223415 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.223401    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:35.225053 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.224752    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:44:35.225713 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.225678    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:44:35.226103 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.225937    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:35.226103 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.226062    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:35.226269 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.226195    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:35.228127 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.228095    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:35.228691 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.228668    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-config-out" (OuterVolumeSpecName: "config-out") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:44:35.228782 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.228722    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-config" (OuterVolumeSpecName: "config") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:35.228847 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.228827    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:35.228906 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.228844    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-kube-api-access-d4xvw" (OuterVolumeSpecName: "kube-api-access-d4xvw") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "kube-api-access-d4xvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:44:35.229026 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.229002    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:35.229095 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.229073    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:35.229187 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.229171    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:35.230183 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.230160    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:35.237506 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.237485    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-web-config" (OuterVolumeSpecName: "web-config") pod "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" (UID: "e3b3aee1-02b2-473d-bda7-b946c4cb5fd0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:35.323921 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.323884    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324017 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.323933    2565 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324017 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.323943    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324017 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.323953    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324017 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.323962    2565 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-grpc-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324017 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.323970    2565 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-kube-rbac-proxy\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324017 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.323978    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324017 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.323988    2565 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-metrics-client-certs\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324017 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.323996    2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-web-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324017 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.324004    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324017 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.324011    2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-config-out\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324017 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.324020    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324344 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.324028    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4xvw\" (UniqueName: \"kubernetes.io/projected/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-kube-api-access-d4xvw\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324344 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.324037    2565 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324344 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.324044    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-prometheus-k8s-db\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324344 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.324053    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-metrics-client-ca\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324344 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.324061    2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-tls-assets\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.324344 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.324068    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:44:35.405714 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.405688    2565 generic.go:356] "Generic (PLEG): container finished" podID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerID="3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb" exitCode=0
Sep 17 16:44:35.405714 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.405707    2565 generic.go:356] "Generic (PLEG): container finished" podID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerID="1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed" exitCode=0
Sep 17 16:44:35.405714 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.405714    2565 generic.go:356] "Generic (PLEG): container finished" podID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerID="8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd" exitCode=0
Sep 17 16:44:35.405856 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.405719    2565 generic.go:356] "Generic (PLEG): container finished" podID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerID="8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3" exitCode=0
Sep 17 16:44:35.405856 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.405724    2565 generic.go:356] "Generic (PLEG): container finished" podID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerID="102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192" exitCode=0
Sep 17 16:44:35.405856 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.405729    2565 generic.go:356] "Generic (PLEG): container finished" podID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerID="1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099" exitCode=0
Sep 17 16:44:35.405856 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.405767    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.405856 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.405775    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0","Type":"ContainerDied","Data":"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb"}
Sep 17 16:44:35.405856 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.405842    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0","Type":"ContainerDied","Data":"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed"}
Sep 17 16:44:35.406059 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.405869    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0","Type":"ContainerDied","Data":"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd"}
Sep 17 16:44:35.406059 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.405895    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0","Type":"ContainerDied","Data":"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3"}
Sep 17 16:44:35.406059 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.405932    2565 scope.go:117] "RemoveContainer" containerID="3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb"
Sep 17 16:44:35.406059 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.405946    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0","Type":"ContainerDied","Data":"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192"}
Sep 17 16:44:35.406059 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.405974    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0","Type":"ContainerDied","Data":"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099"}
Sep 17 16:44:35.406059 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.406006    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3b3aee1-02b2-473d-bda7-b946c4cb5fd0","Type":"ContainerDied","Data":"5bf6a06dd35fd4edc5016bfabe3800e40e851b35375667c3b3c595783915a44c"}
Sep 17 16:44:35.416072 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.416051    2565 scope.go:117] "RemoveContainer" containerID="1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed"
Sep 17 16:44:35.422732 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.422717    2565 scope.go:117] "RemoveContainer" containerID="8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd"
Sep 17 16:44:35.428773 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.428754    2565 scope.go:117] "RemoveContainer" containerID="8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3"
Sep 17 16:44:35.432322 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.432302    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 16:44:35.436026 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.436008    2565 scope.go:117] "RemoveContainer" containerID="102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192"
Sep 17 16:44:35.436637 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.436620    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 16:44:35.443154 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.443138    2565 scope.go:117] "RemoveContainer" containerID="1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099"
Sep 17 16:44:35.449996 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.449976    2565 scope.go:117] "RemoveContainer" containerID="c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea"
Sep 17 16:44:35.456204 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.456189    2565 scope.go:117] "RemoveContainer" containerID="3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb"
Sep 17 16:44:35.456463 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:44:35.456443    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb\": container with ID starting with 3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb not found: ID does not exist" containerID="3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb"
Sep 17 16:44:35.456508 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.456471    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb"} err="failed to get container status \"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb\": rpc error: code = NotFound desc = could not find container \"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb\": container with ID starting with 3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb not found: ID does not exist"
Sep 17 16:44:35.456508 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.456500    2565 scope.go:117] "RemoveContainer" containerID="1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed"
Sep 17 16:44:35.456752 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:44:35.456736    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed\": container with ID starting with 1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed not found: ID does not exist" containerID="1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed"
Sep 17 16:44:35.456796 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.456758    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed"} err="failed to get container status \"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed\": rpc error: code = NotFound desc = could not find container \"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed\": container with ID starting with 1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed not found: ID does not exist"
Sep 17 16:44:35.456796 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.456773    2565 scope.go:117] "RemoveContainer" containerID="8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd"
Sep 17 16:44:35.457041 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:44:35.457023    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd\": container with ID starting with 8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd not found: ID does not exist" containerID="8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd"
Sep 17 16:44:35.457089 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.457046    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd"} err="failed to get container status \"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd\": rpc error: code = NotFound desc = could not find container \"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd\": container with ID starting with 8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd not found: ID does not exist"
Sep 17 16:44:35.457089 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.457060    2565 scope.go:117] "RemoveContainer" containerID="8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3"
Sep 17 16:44:35.457297 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:44:35.457282    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3\": container with ID starting with 8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3 not found: ID does not exist" containerID="8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3"
Sep 17 16:44:35.457333 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.457301    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3"} err="failed to get container status \"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3\": rpc error: code = NotFound desc = could not find container \"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3\": container with ID starting with 8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3 not found: ID does not exist"
Sep 17 16:44:35.457333 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.457319    2565 scope.go:117] "RemoveContainer" containerID="102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192"
Sep 17 16:44:35.457537 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:44:35.457523    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192\": container with ID starting with 102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192 not found: ID does not exist" containerID="102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192"
Sep 17 16:44:35.457574 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.457540    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192"} err="failed to get container status \"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192\": rpc error: code = NotFound desc = could not find container \"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192\": container with ID starting with 102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192 not found: ID does not exist"
Sep 17 16:44:35.457574 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.457552    2565 scope.go:117] "RemoveContainer" containerID="1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099"
Sep 17 16:44:35.457741 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:44:35.457728    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099\": container with ID starting with 1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099 not found: ID does not exist" containerID="1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099"
Sep 17 16:44:35.457777 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.457744    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099"} err="failed to get container status \"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099\": rpc error: code = NotFound desc = could not find container \"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099\": container with ID starting with 1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099 not found: ID does not exist"
Sep 17 16:44:35.457777 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.457756    2565 scope.go:117] "RemoveContainer" containerID="c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea"
Sep 17 16:44:35.457956 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:44:35.457942    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea\": container with ID starting with c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea not found: ID does not exist" containerID="c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea"
Sep 17 16:44:35.458002 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.457959    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea"} err="failed to get container status \"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea\": rpc error: code = NotFound desc = could not find container \"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea\": container with ID starting with c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea not found: ID does not exist"
Sep 17 16:44:35.458002 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.457970    2565 scope.go:117] "RemoveContainer" containerID="3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb"
Sep 17 16:44:35.458201 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.458182    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb"} err="failed to get container status \"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb\": rpc error: code = NotFound desc = could not find container \"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb\": container with ID starting with 3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb not found: ID does not exist"
Sep 17 16:44:35.458258 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.458201    2565 scope.go:117] "RemoveContainer" containerID="1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed"
Sep 17 16:44:35.458451 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.458430    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed"} err="failed to get container status \"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed\": rpc error: code = NotFound desc = could not find container \"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed\": container with ID starting with 1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed not found: ID does not exist"
Sep 17 16:44:35.458530 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.458453    2565 scope.go:117] "RemoveContainer" containerID="8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd"
Sep 17 16:44:35.458702 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.458688    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd"} err="failed to get container status \"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd\": rpc error: code = NotFound desc = could not find container \"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd\": container with ID starting with 8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd not found: ID does not exist"
Sep 17 16:44:35.458749 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.458703    2565 scope.go:117] "RemoveContainer" containerID="8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3"
Sep 17 16:44:35.458965 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.458895    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3"} err="failed to get container status \"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3\": rpc error: code = NotFound desc = could not find container \"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3\": container with ID starting with 8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3 not found: ID does not exist"
Sep 17 16:44:35.458965 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.458932    2565 scope.go:117] "RemoveContainer" containerID="102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192"
Sep 17 16:44:35.459157 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.459139    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192"} err="failed to get container status \"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192\": rpc error: code = NotFound desc = could not find container \"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192\": container with ID starting with 102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192 not found: ID does not exist"
Sep 17 16:44:35.459270 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.459160    2565 scope.go:117] "RemoveContainer" containerID="1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099"
Sep 17 16:44:35.459373 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.459358    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099"} err="failed to get container status \"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099\": rpc error: code = NotFound desc = could not find container \"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099\": container with ID starting with 1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099 not found: ID does not exist"
Sep 17 16:44:35.459415 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.459374    2565 scope.go:117] "RemoveContainer" containerID="c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea"
Sep 17 16:44:35.459583 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.459563    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea"} err="failed to get container status \"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea\": rpc error: code = NotFound desc = could not find container \"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea\": container with ID starting with c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea not found: ID does not exist"
Sep 17 16:44:35.459660 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.459586    2565 scope.go:117] "RemoveContainer" containerID="3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb"
Sep 17 16:44:35.459785 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.459770    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb"} err="failed to get container status \"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb\": rpc error: code = NotFound desc = could not find container \"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb\": container with ID starting with 3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb not found: ID does not exist"
Sep 17 16:44:35.459830 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.459785    2565 scope.go:117] "RemoveContainer" containerID="1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed"
Sep 17 16:44:35.459990 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.459972    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed"} err="failed to get container status \"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed\": rpc error: code = NotFound desc = could not find container \"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed\": container with ID starting with 1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed not found: ID does not exist"
Sep 17 16:44:35.460062 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.459991    2565 scope.go:117] "RemoveContainer" containerID="8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd"
Sep 17 16:44:35.460209 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.460191    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd"} err="failed to get container status \"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd\": rpc error: code = NotFound desc = could not find container \"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd\": container with ID starting with 8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd not found: ID does not exist"
Sep 17 16:44:35.460256 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.460210    2565 scope.go:117] "RemoveContainer" containerID="8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3"
Sep 17 16:44:35.460422 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.460407    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3"} err="failed to get container status \"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3\": rpc error: code = NotFound desc = could not find container \"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3\": container with ID starting with 8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3 not found: ID does not exist"
Sep 17 16:44:35.460462 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.460423    2565 scope.go:117] "RemoveContainer" containerID="102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192"
Sep 17 16:44:35.460689 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.460622    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192"} err="failed to get container status \"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192\": rpc error: code = NotFound desc = could not find container \"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192\": container with ID starting with 102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192 not found: ID does not exist"
Sep 17 16:44:35.460770 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.460677    2565 scope.go:117] "RemoveContainer" containerID="1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099"
Sep 17 16:44:35.461222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.461198    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099"} err="failed to get container status \"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099\": rpc error: code = NotFound desc = could not find container \"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099\": container with ID starting with 1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099 not found: ID does not exist"
Sep 17 16:44:35.461222 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.461221    2565 scope.go:117] "RemoveContainer" containerID="c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea"
Sep 17 16:44:35.461500 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.461480    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea"} err="failed to get container status \"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea\": rpc error: code = NotFound desc = could not find container \"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea\": container with ID starting with c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea not found: ID does not exist"
Sep 17 16:44:35.461586 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.461502    2565 scope.go:117] "RemoveContainer" containerID="3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb"
Sep 17 16:44:35.461803 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.461780    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb"} err="failed to get container status \"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb\": rpc error: code = NotFound desc = could not find container \"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb\": container with ID starting with 3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb not found: ID does not exist"
Sep 17 16:44:35.461878 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.461804    2565 scope.go:117] "RemoveContainer" containerID="1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed"
Sep 17 16:44:35.462092 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462075    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed"} err="failed to get container status \"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed\": rpc error: code = NotFound desc = could not find container \"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed\": container with ID starting with 1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed not found: ID does not exist"
Sep 17 16:44:35.462156 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462092    2565 scope.go:117] "RemoveContainer" containerID="8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd"
Sep 17 16:44:35.462333 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462316    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 16:44:35.462378 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462327    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd"} err="failed to get container status \"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd\": rpc error: code = NotFound desc = could not find container \"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd\": container with ID starting with 8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd not found: ID does not exist"
Sep 17 16:44:35.462378 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462354    2565 scope.go:117] "RemoveContainer" containerID="8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3"
Sep 17 16:44:35.462592 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462579    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="thanos-sidecar"
Sep 17 16:44:35.462592 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462592    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="thanos-sidecar"
Sep 17 16:44:35.462682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462591    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3"} err="failed to get container status \"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3\": rpc error: code = NotFound desc = could not find container \"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3\": container with ID starting with 8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3 not found: ID does not exist"
Sep 17 16:44:35.462682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462601    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="kube-rbac-proxy-thanos"
Sep 17 16:44:35.462682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462604    2565 scope.go:117] "RemoveContainer" containerID="102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192"
Sep 17 16:44:35.462682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462607    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="kube-rbac-proxy-thanos"
Sep 17 16:44:35.462682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462614    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="init-config-reloader"
Sep 17 16:44:35.462682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462619    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="init-config-reloader"
Sep 17 16:44:35.462682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462625    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="config-reloader"
Sep 17 16:44:35.462682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462630    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="config-reloader"
Sep 17 16:44:35.462682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462636    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="prometheus"
Sep 17 16:44:35.462682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462641    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="prometheus"
Sep 17 16:44:35.462682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462652    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="kube-rbac-proxy-web"
Sep 17 16:44:35.462682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462658    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="kube-rbac-proxy-web"
Sep 17 16:44:35.462682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462663    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="kube-rbac-proxy"
Sep 17 16:44:35.462682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462668    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="kube-rbac-proxy"
Sep 17 16:44:35.463159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462711    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="kube-rbac-proxy-thanos"
Sep 17 16:44:35.463159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462718    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="config-reloader"
Sep 17 16:44:35.463159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462724    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="prometheus"
Sep 17 16:44:35.463159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462730    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="kube-rbac-proxy-web"
Sep 17 16:44:35.463159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462735    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="kube-rbac-proxy"
Sep 17 16:44:35.463159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462741    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" containerName="thanos-sidecar"
Sep 17 16:44:35.463159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462833    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192"} err="failed to get container status \"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192\": rpc error: code = NotFound desc = could not find container \"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192\": container with ID starting with 102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192 not found: ID does not exist"
Sep 17 16:44:35.463159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.462847    2565 scope.go:117] "RemoveContainer" containerID="1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099"
Sep 17 16:44:35.463159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.463091    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099"} err="failed to get container status \"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099\": rpc error: code = NotFound desc = could not find container \"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099\": container with ID starting with 1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099 not found: ID does not exist"
Sep 17 16:44:35.463159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.463108    2565 scope.go:117] "RemoveContainer" containerID="c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea"
Sep 17 16:44:35.463477 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.463346    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea"} err="failed to get container status \"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea\": rpc error: code = NotFound desc = could not find container \"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea\": container with ID starting with c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea not found: ID does not exist"
Sep 17 16:44:35.463477 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.463362    2565 scope.go:117] "RemoveContainer" containerID="3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb"
Sep 17 16:44:35.463601 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.463581    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb"} err="failed to get container status \"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb\": rpc error: code = NotFound desc = could not find container \"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb\": container with ID starting with 3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb not found: ID does not exist"
Sep 17 16:44:35.463664 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.463604    2565 scope.go:117] "RemoveContainer" containerID="1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed"
Sep 17 16:44:35.463848 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.463830    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed"} err="failed to get container status \"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed\": rpc error: code = NotFound desc = could not find container \"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed\": container with ID starting with 1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed not found: ID does not exist"
Sep 17 16:44:35.463889 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.463850    2565 scope.go:117] "RemoveContainer" containerID="8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd"
Sep 17 16:44:35.464100 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.464084    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd"} err="failed to get container status \"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd\": rpc error: code = NotFound desc = could not find container \"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd\": container with ID starting with 8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd not found: ID does not exist"
Sep 17 16:44:35.464169 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.464101    2565 scope.go:117] "RemoveContainer" containerID="8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3"
Sep 17 16:44:35.464322 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.464304    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3"} err="failed to get container status \"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3\": rpc error: code = NotFound desc = could not find container \"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3\": container with ID starting with 8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3 not found: ID does not exist"
Sep 17 16:44:35.464388 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.464322    2565 scope.go:117] "RemoveContainer" containerID="102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192"
Sep 17 16:44:35.464548 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.464528    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192"} err="failed to get container status \"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192\": rpc error: code = NotFound desc = could not find container \"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192\": container with ID starting with 102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192 not found: ID does not exist"
Sep 17 16:44:35.464605 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.464548    2565 scope.go:117] "RemoveContainer" containerID="1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099"
Sep 17 16:44:35.464761 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.464743    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099"} err="failed to get container status \"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099\": rpc error: code = NotFound desc = could not find container \"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099\": container with ID starting with 1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099 not found: ID does not exist"
Sep 17 16:44:35.464800 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.464762    2565 scope.go:117] "RemoveContainer" containerID="c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea"
Sep 17 16:44:35.465044 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.465028    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea"} err="failed to get container status \"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea\": rpc error: code = NotFound desc = could not find container \"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea\": container with ID starting with c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea not found: ID does not exist"
Sep 17 16:44:35.465044 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.465046    2565 scope.go:117] "RemoveContainer" containerID="3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb"
Sep 17 16:44:35.465270 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.465253    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb"} err="failed to get container status \"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb\": rpc error: code = NotFound desc = could not find container \"3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb\": container with ID starting with 3d975f1b04a08b129d482801f05e7ffc40f7be3b44e7944af9468438f9a790bb not found: ID does not exist"
Sep 17 16:44:35.465311 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.465272    2565 scope.go:117] "RemoveContainer" containerID="1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed"
Sep 17 16:44:35.465495 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.465478    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed"} err="failed to get container status \"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed\": rpc error: code = NotFound desc = could not find container \"1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed\": container with ID starting with 1c9bb2531f3e22e0c187faa369d45ff31ef2095e40f9aacf2c0107807af858ed not found: ID does not exist"
Sep 17 16:44:35.465747 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.465496    2565 scope.go:117] "RemoveContainer" containerID="8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd"
Sep 17 16:44:35.465747 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.465721    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd"} err="failed to get container status \"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd\": rpc error: code = NotFound desc = could not find container \"8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd\": container with ID starting with 8e42f2d51b7a764a6f932de81c3038f954dfd88703032a9a3628dfb543d7aadd not found: ID does not exist"
Sep 17 16:44:35.465747 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.465740    2565 scope.go:117] "RemoveContainer" containerID="8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3"
Sep 17 16:44:35.466029 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.466005    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3"} err="failed to get container status \"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3\": rpc error: code = NotFound desc = could not find container \"8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3\": container with ID starting with 8f0f3ed43a66ee8fb489e015fcb1874bccd90c1dfe8164b2091586072685b2c3 not found: ID does not exist"
Sep 17 16:44:35.466090 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.466031    2565 scope.go:117] "RemoveContainer" containerID="102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192"
Sep 17 16:44:35.466194 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.466180    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.466317 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.466297    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192"} err="failed to get container status \"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192\": rpc error: code = NotFound desc = could not find container \"102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192\": container with ID starting with 102f66c8b6ff0ce9cf0dd0eb6a35d27184c66a01cdf1a4c0ce3ff9921e2cc192 not found: ID does not exist"
Sep 17 16:44:35.466366 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.466319    2565 scope.go:117] "RemoveContainer" containerID="1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099"
Sep 17 16:44:35.466610 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.466576    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099"} err="failed to get container status \"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099\": rpc error: code = NotFound desc = could not find container \"1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099\": container with ID starting with 1643646fbcd50c421ed78f5f6066f0905a9bfdbcb89b929c7533879d7652e099 not found: ID does not exist"
Sep 17 16:44:35.466610 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.466600    2565 scope.go:117] "RemoveContainer" containerID="c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea"
Sep 17 16:44:35.466937 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.466897    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea"} err="failed to get container status \"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea\": rpc error: code = NotFound desc = could not find container \"c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea\": container with ID starting with c91c0381ca7547a9b044764d0d82b6752daabbd93090f3be851ab1f30a3fe3ea not found: ID does not exist"
Sep 17 16:44:35.468807 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.468791    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 16:44:35.469013 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.468998    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 16:44:35.469276 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.469258    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 16:44:35.469358 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.469291    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 16:44:35.469358 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.469309    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 16:44:35.469358 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.469259    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 16:44:35.469522 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.469392    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qvc78\""
Sep 17 16:44:35.469522 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.469418    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 16:44:35.469688 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.469673    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 16:44:35.470003 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.469986    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 16:44:35.470137 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.469997    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4bocskh2vfiqn\""
Sep 17 16:44:35.470212 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.470004    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 16:44:35.470212 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.470025    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 16:44:35.470212 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.470045    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 16:44:35.471746 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.471726    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 16:44:35.474023 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.474004    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 16:44:35.485560 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.485537    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 16:44:35.525996 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.525971    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526096 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526096 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526218 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526093    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526218 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526146    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526218 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526178    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-web-config\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526218 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526201    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526386 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526230    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-config-out\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526386 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526254    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526386 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526386 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526307    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526386 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526366    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526556 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526390    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526556 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526556 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526450    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526556 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526472    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-config\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526556 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526487    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr2zx\" (UniqueName: \"kubernetes.io/projected/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-kube-api-access-hr2zx\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526556 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526503    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526556 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526517    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.526556 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.526546    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627210 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627308 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627308 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627247    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627308 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627276    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627308 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627301    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627483 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627328    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-web-config\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627483 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627483 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627375    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-config-out\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627483 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627477    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627683 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627502    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627683 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627535    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627683 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627573    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627683 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627598    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627683 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627629    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627683 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627658    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627988 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-config\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627988 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627712    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627988 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627720    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr2zx\" (UniqueName: \"kubernetes.io/projected/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-kube-api-access-hr2zx\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627988 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627760    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627988 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627791    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627988 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627988 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627960    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.627988 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.627985    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.628343 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.628035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.628626 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.628604    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.628799 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.628775    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.629505 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.629478    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.630334 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.630150    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.630334 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.630287    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-web-config\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.631011 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.630699    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.631011 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.630903    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.631011 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.630961    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.631228 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.631209    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.631467 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.631443    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.632220 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.632200    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-config-out\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.632484 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.632466    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.632760 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.632744    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.633043 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.633023    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-config\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.633599 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.633583    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.635538 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.635522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr2zx\" (UniqueName: \"kubernetes.io/projected/02f0ef06-ac87-4d72-a9ba-70f8fdbd311a-kube-api-access-hr2zx\") pod \"prometheus-k8s-0\" (UID: \"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:35.775596 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:35.775547    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:36.053887 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:36.052262    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 16:44:36.055173 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:44:36.055145    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f0ef06_ac87_4d72_a9ba_70f8fdbd311a.slice/crio-55b4b00ece1a6eeba79e8842c69240ac829d433c5c1540786d9136d2fa5a3393 WatchSource:0}: Error finding container 55b4b00ece1a6eeba79e8842c69240ac829d433c5c1540786d9136d2fa5a3393: Status 404 returned error can't find the container with id 55b4b00ece1a6eeba79e8842c69240ac829d433c5c1540786d9136d2fa5a3393
Sep 17 16:44:36.411371 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:36.411333    2565 generic.go:356] "Generic (PLEG): container finished" podID="02f0ef06-ac87-4d72-a9ba-70f8fdbd311a" containerID="615ce643a03c2d918025e1bdefd14ae56024bc1fa4496ecfc34084078c60b082" exitCode=0
Sep 17 16:44:36.411476 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:36.411428    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a","Type":"ContainerDied","Data":"615ce643a03c2d918025e1bdefd14ae56024bc1fa4496ecfc34084078c60b082"}
Sep 17 16:44:36.411612 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:36.411527    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a","Type":"ContainerStarted","Data":"55b4b00ece1a6eeba79e8842c69240ac829d433c5c1540786d9136d2fa5a3393"}
Sep 17 16:44:36.822714 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:36.822685    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3b3aee1-02b2-473d-bda7-b946c4cb5fd0" path="/var/lib/kubelet/pods/e3b3aee1-02b2-473d-bda7-b946c4cb5fd0/volumes"
Sep 17 16:44:37.419190 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:37.419160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a","Type":"ContainerStarted","Data":"bc60ed83439869edac08322b045b9067bd8cdedb448eaf49beda59a108af18c3"}
Sep 17 16:44:37.419514 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:37.419223    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a","Type":"ContainerStarted","Data":"23558be4dc0e7c5b7b409ee876a01d56f96ef0d80295cfd2ef28bfedada3c7fd"}
Sep 17 16:44:37.419514 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:37.419248    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a","Type":"ContainerStarted","Data":"5764289b70edced404bd3aa901799bd76010841131251afff548ba77b3f3ab5a"}
Sep 17 16:44:37.419514 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:37.419274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a","Type":"ContainerStarted","Data":"63bb273420869e689bf69c82808bb8b54c88d7597070cc1102fe76f2eca1de4d"}
Sep 17 16:44:37.419514 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:37.419298    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a","Type":"ContainerStarted","Data":"baa5ae886d6046d362403ea33f8c1202ada4e554ca731d964edfeafd55695a76"}
Sep 17 16:44:37.419514 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:37.419323    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"02f0ef06-ac87-4d72-a9ba-70f8fdbd311a","Type":"ContainerStarted","Data":"f617c01e148bb6960338ec4d9a61b275d6b5db62ee4357667659f34e29a7be32"}
Sep 17 16:44:37.449096 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:37.449041    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.428998986 podStartE2EDuration="2.449019806s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.02082ms" isStatefulPod=false podCreationTimestamp="2026-09-17 16:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:44:37.446216602 +0000 UTC m=+143.012328336" watchObservedRunningTime="2026-09-17 16:44:37.449019806 +0000 UTC m=+143.015131529"
Sep 17 16:44:38.178194 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:38.178159    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:40.776551 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:40.776510    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:44:49.201194 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:49.201146    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:49.252795 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:49.252771    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:44:49.307411 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:44:49.307388    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:01.319547 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:45:01.319456    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:15.803248 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:45:15.803215    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:15.903811 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:45:15.903778    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:35.776040 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:45:35.776009    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:45:35.791532 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:45:35.791508    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:45:36.651297 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:45:36.651267    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 16:45:44.854523 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:45:44.854491    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:44.919771 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:45:44.919745    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:52.418834 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:45:52.418798    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:56.690421 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:45:56.690391    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:45:56.744659 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:45:56.744621    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:03.783513 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:46:03.783455    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:03.842987 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:46:03.842946    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:10.674244 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:46:10.674214    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:10.857788 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:46:10.857766    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:10.914904 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:46:10.914875    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:53.759395 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:46:53.759286    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:46:55.279492 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:46:55.279434    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:16.709319 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:47:16.709294    2565 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 16:47:21.030538 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:47:21.030478    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:21.135908 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:47:21.135876    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:26.679792 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:47:26.679743    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:26.980276 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:47:26.980190    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:27.203012 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:47:27.202979    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:27.402094 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:47:27.402066    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:27.571154 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:47:27.571133    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:27.736953 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:47:27.736866    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:27.834452 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:47:27.834417    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:27.934008 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:47:27.933972    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:28.140642 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:47:28.140618    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:28.239228 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:47:28.239204    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:28.440557 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:47:28.440464    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:28.540122 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:47:28.540096    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:47:28.661362 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:47:28.661338    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:12.350409 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:12.350371    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:19.578206 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:19.578175    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:19.632170 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:19.632133    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:26.142365 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:26.142295    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:26.273958 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:26.273908    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:37.134020 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:37.133990    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:37.234723 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:37.234665    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:37.295104 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:37.295080    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:40.898483 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:40.898434    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:47.391433 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.391392    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:47.472869 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.472840    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:47.493828 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.493801    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-6tlkz"]
Sep 17 16:48:47.496983 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.496964    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-6tlkz"
Sep 17 16:48:47.499498 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.499477    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 16:48:47.500359 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.500332    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 16:48:47.500359 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.500344    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 16:48:47.500359 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.500357    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-pqs8n\""
Sep 17 16:48:47.506065 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.506046    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-6tlkz"]
Sep 17 16:48:47.549346 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.549328    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:47.619263 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.619243    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t98gb\" (UniqueName: \"kubernetes.io/projected/da80d565-728c-4e46-b0a8-330443915b96-kube-api-access-t98gb\") pod \"seaweedfs-64568bcd49-6tlkz\" (UID: \"da80d565-728c-4e46-b0a8-330443915b96\") " pod="kserve/seaweedfs-64568bcd49-6tlkz"
Sep 17 16:48:47.619339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.619280    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/da80d565-728c-4e46-b0a8-330443915b96-data\") pod \"seaweedfs-64568bcd49-6tlkz\" (UID: \"da80d565-728c-4e46-b0a8-330443915b96\") " pod="kserve/seaweedfs-64568bcd49-6tlkz"
Sep 17 16:48:47.707935 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.707850    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:47.719787 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.719765    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t98gb\" (UniqueName: \"kubernetes.io/projected/da80d565-728c-4e46-b0a8-330443915b96-kube-api-access-t98gb\") pod \"seaweedfs-64568bcd49-6tlkz\" (UID: \"da80d565-728c-4e46-b0a8-330443915b96\") " pod="kserve/seaweedfs-64568bcd49-6tlkz"
Sep 17 16:48:47.719883 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.719808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/da80d565-728c-4e46-b0a8-330443915b96-data\") pod \"seaweedfs-64568bcd49-6tlkz\" (UID: \"da80d565-728c-4e46-b0a8-330443915b96\") " pod="kserve/seaweedfs-64568bcd49-6tlkz"
Sep 17 16:48:47.720148 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.720131    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/da80d565-728c-4e46-b0a8-330443915b96-data\") pod \"seaweedfs-64568bcd49-6tlkz\" (UID: \"da80d565-728c-4e46-b0a8-330443915b96\") " pod="kserve/seaweedfs-64568bcd49-6tlkz"
Sep 17 16:48:47.727696 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.727671    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t98gb\" (UniqueName: \"kubernetes.io/projected/da80d565-728c-4e46-b0a8-330443915b96-kube-api-access-t98gb\") pod \"seaweedfs-64568bcd49-6tlkz\" (UID: \"da80d565-728c-4e46-b0a8-330443915b96\") " pod="kserve/seaweedfs-64568bcd49-6tlkz"
Sep 17 16:48:47.805636 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.805605    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-6tlkz"
Sep 17 16:48:47.809262 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.809244    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:47.965758 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:47.965691    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:48.063254 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:48.063231    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:48.073564 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:48.073544    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-6tlkz"]
Sep 17 16:48:48.075790 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:48:48.075758    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda80d565_728c_4e46_b0a8_330443915b96.slice/crio-2df78fe379002a8c059408574500d666e731767e180bc73bda1bf5f180987474 WatchSource:0}: Error finding container 2df78fe379002a8c059408574500d666e731767e180bc73bda1bf5f180987474: Status 404 returned error can't find the container with id 2df78fe379002a8c059408574500d666e731767e180bc73bda1bf5f180987474
Sep 17 16:48:48.076932 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:48.076894    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:48:48.114213 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:48.114194    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:48:48.313240 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:48.313188    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-6tlkz" event={"ID":"da80d565-728c-4e46-b0a8-330443915b96","Type":"ContainerStarted","Data":"2df78fe379002a8c059408574500d666e731767e180bc73bda1bf5f180987474"}
Sep 17 16:48:51.326369 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:51.326303    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-6tlkz" event={"ID":"da80d565-728c-4e46-b0a8-330443915b96","Type":"ContainerStarted","Data":"0884860783b84c36f01c762eacae23d61659c2e03de794b7909eed88ece17b11"}
Sep 17 16:48:51.326694 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:51.326418    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-6tlkz"
Sep 17 16:48:51.342185 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:51.342142    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-6tlkz" podStartSLOduration=1.443293487 podStartE2EDuration="4.342129098s" totalImagesPullingTime="2.898835611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:48:51.341535761 +0000 UTC m=+396.907647485" watchObservedRunningTime="2026-09-17 16:48:51.342129098 +0000 UTC m=+396.908240820"
Sep 17 16:48:57.332566 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:48:57.332537    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-6tlkz"
Sep 17 16:49:38.301485 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:49:38.301394    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:50:04.454257 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.454209    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:50:04.507309 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.507283    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-9sx8z"]
Sep 17 16:50:04.510384 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.510364    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-9sx8z"
Sep 17 16:50:04.512810 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.512779    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 16:50:04.512810 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.512788    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 16:50:04.512979 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.512817    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-kzpbd\""
Sep 17 16:50:04.520632 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.520604    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-9sx8z"]
Sep 17 16:50:04.527976 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.527954    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:50:04.638389 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.638365    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186d3f5e-1136-43b6-a41a-4750dd3e1a65-metrics-certs\") pod \"odh-model-controller-7b966d7984-9sx8z\" (UID: \"186d3f5e-1136-43b6-a41a-4750dd3e1a65\") " pod="kserve/odh-model-controller-7b966d7984-9sx8z"
Sep 17 16:50:04.638507 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.638413    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/186d3f5e-1136-43b6-a41a-4750dd3e1a65-cert\") pod \"odh-model-controller-7b966d7984-9sx8z\" (UID: \"186d3f5e-1136-43b6-a41a-4750dd3e1a65\") " pod="kserve/odh-model-controller-7b966d7984-9sx8z"
Sep 17 16:50:04.638507 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.638449    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxzmz\" (UniqueName: \"kubernetes.io/projected/186d3f5e-1136-43b6-a41a-4750dd3e1a65-kube-api-access-fxzmz\") pod \"odh-model-controller-7b966d7984-9sx8z\" (UID: \"186d3f5e-1136-43b6-a41a-4750dd3e1a65\") " pod="kserve/odh-model-controller-7b966d7984-9sx8z"
Sep 17 16:50:04.738966 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.738886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxzmz\" (UniqueName: \"kubernetes.io/projected/186d3f5e-1136-43b6-a41a-4750dd3e1a65-kube-api-access-fxzmz\") pod \"odh-model-controller-7b966d7984-9sx8z\" (UID: \"186d3f5e-1136-43b6-a41a-4750dd3e1a65\") " pod="kserve/odh-model-controller-7b966d7984-9sx8z"
Sep 17 16:50:04.739062 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.738997    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186d3f5e-1136-43b6-a41a-4750dd3e1a65-metrics-certs\") pod \"odh-model-controller-7b966d7984-9sx8z\" (UID: \"186d3f5e-1136-43b6-a41a-4750dd3e1a65\") " pod="kserve/odh-model-controller-7b966d7984-9sx8z"
Sep 17 16:50:04.739062 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.739039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/186d3f5e-1136-43b6-a41a-4750dd3e1a65-cert\") pod \"odh-model-controller-7b966d7984-9sx8z\" (UID: \"186d3f5e-1136-43b6-a41a-4750dd3e1a65\") " pod="kserve/odh-model-controller-7b966d7984-9sx8z"
Sep 17 16:50:04.739168 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:50:04.739126    2565 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 17 16:50:04.739168 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:50:04.739140    2565 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 16:50:04.739251 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:50:04.739188    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/186d3f5e-1136-43b6-a41a-4750dd3e1a65-metrics-certs podName:186d3f5e-1136-43b6-a41a-4750dd3e1a65 nodeName:}" failed. No retries permitted until 2026-09-17 16:50:05.239173608 +0000 UTC m=+470.805285307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/186d3f5e-1136-43b6-a41a-4750dd3e1a65-metrics-certs") pod "odh-model-controller-7b966d7984-9sx8z" (UID: "186d3f5e-1136-43b6-a41a-4750dd3e1a65") : secret "odh-model-controller-metrics-tls" not found
Sep 17 16:50:04.739251 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:50:04.739200    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/186d3f5e-1136-43b6-a41a-4750dd3e1a65-cert podName:186d3f5e-1136-43b6-a41a-4750dd3e1a65 nodeName:}" failed. No retries permitted until 2026-09-17 16:50:05.239195102 +0000 UTC m=+470.805306802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/186d3f5e-1136-43b6-a41a-4750dd3e1a65-cert") pod "odh-model-controller-7b966d7984-9sx8z" (UID: "186d3f5e-1136-43b6-a41a-4750dd3e1a65") : secret "odh-model-controller-webhook-cert" not found
Sep 17 16:50:04.744753 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.744728    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:50:04.747288 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.747267    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxzmz\" (UniqueName: \"kubernetes.io/projected/186d3f5e-1136-43b6-a41a-4750dd3e1a65-kube-api-access-fxzmz\") pod \"odh-model-controller-7b966d7984-9sx8z\" (UID: \"186d3f5e-1136-43b6-a41a-4750dd3e1a65\") " pod="kserve/odh-model-controller-7b966d7984-9sx8z"
Sep 17 16:50:04.878736 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.878715    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:50:04.980036 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:04.980015    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:50:05.243719 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:05.243697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186d3f5e-1136-43b6-a41a-4750dd3e1a65-metrics-certs\") pod \"odh-model-controller-7b966d7984-9sx8z\" (UID: \"186d3f5e-1136-43b6-a41a-4750dd3e1a65\") " pod="kserve/odh-model-controller-7b966d7984-9sx8z"
Sep 17 16:50:05.243841 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:05.243729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/186d3f5e-1136-43b6-a41a-4750dd3e1a65-cert\") pod \"odh-model-controller-7b966d7984-9sx8z\" (UID: \"186d3f5e-1136-43b6-a41a-4750dd3e1a65\") " pod="kserve/odh-model-controller-7b966d7984-9sx8z"
Sep 17 16:50:05.245892 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:05.245871    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186d3f5e-1136-43b6-a41a-4750dd3e1a65-metrics-certs\") pod \"odh-model-controller-7b966d7984-9sx8z\" (UID: \"186d3f5e-1136-43b6-a41a-4750dd3e1a65\") " pod="kserve/odh-model-controller-7b966d7984-9sx8z"
Sep 17 16:50:05.246063 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:05.246045    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/186d3f5e-1136-43b6-a41a-4750dd3e1a65-cert\") pod \"odh-model-controller-7b966d7984-9sx8z\" (UID: \"186d3f5e-1136-43b6-a41a-4750dd3e1a65\") " pod="kserve/odh-model-controller-7b966d7984-9sx8z"
Sep 17 16:50:05.421025 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:05.420995    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-9sx8z"
Sep 17 16:50:05.691137 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:05.691114    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-9sx8z"]
Sep 17 16:50:05.693490 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:50:05.693467    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186d3f5e_1136_43b6_a41a_4750dd3e1a65.slice/crio-37c2cfb594c3d4574c743ac4c7598faa1b9622006cd0431fd23dd4d4fef2aeab WatchSource:0}: Error finding container 37c2cfb594c3d4574c743ac4c7598faa1b9622006cd0431fd23dd4d4fef2aeab: Status 404 returned error can't find the container with id 37c2cfb594c3d4574c743ac4c7598faa1b9622006cd0431fd23dd4d4fef2aeab
Sep 17 16:50:05.825514 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:05.825492    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:50:05.926806 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:05.926785    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:50:06.608362 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:06.608235    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-9sx8z" event={"ID":"186d3f5e-1136-43b6-a41a-4750dd3e1a65","Type":"ContainerStarted","Data":"37c2cfb594c3d4574c743ac4c7598faa1b9622006cd0431fd23dd4d4fef2aeab"}
Sep 17 16:50:08.618330 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:08.618296    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-9sx8z" event={"ID":"186d3f5e-1136-43b6-a41a-4750dd3e1a65","Type":"ContainerStarted","Data":"7eb3998f4695bf3b3f37b98fececd66ec70af91e72612fa8bb18704d299b3ffb"}
Sep 17 16:50:08.618721 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:08.618422    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-9sx8z"
Sep 17 16:50:08.635503 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:08.635456    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-9sx8z" podStartSLOduration=2.29231413 podStartE2EDuration="4.635444117s" totalImagesPullingTime="2.343129987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:50:08.634803804 +0000 UTC m=+474.200915528" watchObservedRunningTime="2026-09-17 16:50:08.635444117 +0000 UTC m=+474.201555838"
Sep 17 16:50:19.624791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:19.624757    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-9sx8z"
Sep 17 16:50:20.434375 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:20.434340    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-mq9nl"]
Sep 17 16:50:20.437903 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:20.437885    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mq9nl"
Sep 17 16:50:20.440249 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:20.440230    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 16:50:20.445900 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:20.445879    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-mq9nl"]
Sep 17 16:50:20.453252 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:20.453232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e860760-2ab7-4db2-80af-91def4cdc1dd-model-cache\") pod \"s3-init-mq9nl\" (UID: \"5e860760-2ab7-4db2-80af-91def4cdc1dd\") " pod="kserve/s3-init-mq9nl"
Sep 17 16:50:20.453358 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:20.453270    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk7p2\" (UniqueName: \"kubernetes.io/projected/5e860760-2ab7-4db2-80af-91def4cdc1dd-kube-api-access-fk7p2\") pod \"s3-init-mq9nl\" (UID: \"5e860760-2ab7-4db2-80af-91def4cdc1dd\") " pod="kserve/s3-init-mq9nl"
Sep 17 16:50:20.554390 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:20.554366    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e860760-2ab7-4db2-80af-91def4cdc1dd-model-cache\") pod \"s3-init-mq9nl\" (UID: \"5e860760-2ab7-4db2-80af-91def4cdc1dd\") " pod="kserve/s3-init-mq9nl"
Sep 17 16:50:20.554477 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:20.554398    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk7p2\" (UniqueName: \"kubernetes.io/projected/5e860760-2ab7-4db2-80af-91def4cdc1dd-kube-api-access-fk7p2\") pod \"s3-init-mq9nl\" (UID: \"5e860760-2ab7-4db2-80af-91def4cdc1dd\") " pod="kserve/s3-init-mq9nl"
Sep 17 16:50:20.554784 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:20.554768    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e860760-2ab7-4db2-80af-91def4cdc1dd-model-cache\") pod \"s3-init-mq9nl\" (UID: \"5e860760-2ab7-4db2-80af-91def4cdc1dd\") " pod="kserve/s3-init-mq9nl"
Sep 17 16:50:20.563162 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:20.563136    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk7p2\" (UniqueName: \"kubernetes.io/projected/5e860760-2ab7-4db2-80af-91def4cdc1dd-kube-api-access-fk7p2\") pod \"s3-init-mq9nl\" (UID: \"5e860760-2ab7-4db2-80af-91def4cdc1dd\") " pod="kserve/s3-init-mq9nl"
Sep 17 16:50:20.755890 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:20.755838    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mq9nl"
Sep 17 16:50:21.025196 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:21.025128    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-mq9nl"]
Sep 17 16:50:21.028045 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:50:21.028020    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e860760_2ab7_4db2_80af_91def4cdc1dd.slice/crio-2e197c3948136cfa3631d589fc6d589a9e0905093ed3c8adbc5bbffda3c52614 WatchSource:0}: Error finding container 2e197c3948136cfa3631d589fc6d589a9e0905093ed3c8adbc5bbffda3c52614: Status 404 returned error can't find the container with id 2e197c3948136cfa3631d589fc6d589a9e0905093ed3c8adbc5bbffda3c52614
Sep 17 16:50:21.671070 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:21.671033    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mq9nl" event={"ID":"5e860760-2ab7-4db2-80af-91def4cdc1dd","Type":"ContainerStarted","Data":"2e197c3948136cfa3631d589fc6d589a9e0905093ed3c8adbc5bbffda3c52614"}
Sep 17 16:50:24.687270 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:24.687235    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mq9nl" event={"ID":"5e860760-2ab7-4db2-80af-91def4cdc1dd","Type":"ContainerStarted","Data":"0ef6c5270870c5ddc07cb8032690fe2c0fc0c81bcb0c6da29d57f2d101bda43a"}
Sep 17 16:50:26.452035 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:26.452007    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:50:26.553284 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:26.553219    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:50:26.696312 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:26.696286    2565 generic.go:356] "Generic (PLEG): container finished" podID="5e860760-2ab7-4db2-80af-91def4cdc1dd" containerID="0ef6c5270870c5ddc07cb8032690fe2c0fc0c81bcb0c6da29d57f2d101bda43a" exitCode=0
Sep 17 16:50:26.696418 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:26.696360    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mq9nl" event={"ID":"5e860760-2ab7-4db2-80af-91def4cdc1dd","Type":"ContainerDied","Data":"0ef6c5270870c5ddc07cb8032690fe2c0fc0c81bcb0c6da29d57f2d101bda43a"}
Sep 17 16:50:31.726529 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:31.726488    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mq9nl" event={"ID":"5e860760-2ab7-4db2-80af-91def4cdc1dd","Type":"ContainerStarted","Data":"61f582e0ce845c02e92f0317d8326d9e3d3ccb2254d5a1f0bd1c75ae36d2c8c4"}
Sep 17 16:50:31.741691 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:50:31.741649    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-mq9nl" podStartSLOduration=1.993842161 podStartE2EDuration="11.741634764s" totalImagesPullingTime="7.495044896s" totalInitContainerRuntime="2.252747707s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:50:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:50:31.740603406 +0000 UTC m=+497.306715131" watchObservedRunningTime="2026-09-17 16:50:31.741634764 +0000 UTC m=+497.307746485"
Sep 17 16:51:29.440285 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:29.440233    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:51:33.514394 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:33.514355    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:51:35.818563 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:35.818530    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/s3-init-mq9nl" podUID="5e860760-2ab7-4db2-80af-91def4cdc1dd" containerName="s3-init" containerID="cri-o://61f582e0ce845c02e92f0317d8326d9e3d3ccb2254d5a1f0bd1c75ae36d2c8c4" gracePeriod=30
Sep 17 16:51:35.964998 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:35.964973    2565 generic.go:356] "Generic (PLEG): container finished" podID="5e860760-2ab7-4db2-80af-91def4cdc1dd" containerID="61f582e0ce845c02e92f0317d8326d9e3d3ccb2254d5a1f0bd1c75ae36d2c8c4" exitCode=0
Sep 17 16:51:35.965102 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:35.965046    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mq9nl" event={"ID":"5e860760-2ab7-4db2-80af-91def4cdc1dd","Type":"ContainerDied","Data":"61f582e0ce845c02e92f0317d8326d9e3d3ccb2254d5a1f0bd1c75ae36d2c8c4"}
Sep 17 16:51:35.965102 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:35.965086    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mq9nl" event={"ID":"5e860760-2ab7-4db2-80af-91def4cdc1dd","Type":"ContainerDied","Data":"2e197c3948136cfa3631d589fc6d589a9e0905093ed3c8adbc5bbffda3c52614"}
Sep 17 16:51:35.965102 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:35.965097    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e197c3948136cfa3631d589fc6d589a9e0905093ed3c8adbc5bbffda3c52614"
Sep 17 16:51:35.965196 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:35.965152    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mq9nl"
Sep 17 16:51:36.967926 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:36.967877    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mq9nl"
Sep 17 16:51:36.974372 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:36.974352    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mq9nl"
Sep 17 16:51:37.109142 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:37.109122    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e860760-2ab7-4db2-80af-91def4cdc1dd-model-cache\") pod \"5e860760-2ab7-4db2-80af-91def4cdc1dd\" (UID: \"5e860760-2ab7-4db2-80af-91def4cdc1dd\") "
Sep 17 16:51:37.109233 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:37.109166    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk7p2\" (UniqueName: \"kubernetes.io/projected/5e860760-2ab7-4db2-80af-91def4cdc1dd-kube-api-access-fk7p2\") pod \"5e860760-2ab7-4db2-80af-91def4cdc1dd\" (UID: \"5e860760-2ab7-4db2-80af-91def4cdc1dd\") "
Sep 17 16:51:37.111282 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:37.111259    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e860760-2ab7-4db2-80af-91def4cdc1dd-kube-api-access-fk7p2" (OuterVolumeSpecName: "kube-api-access-fk7p2") pod "5e860760-2ab7-4db2-80af-91def4cdc1dd" (UID: "5e860760-2ab7-4db2-80af-91def4cdc1dd"). InnerVolumeSpecName "kube-api-access-fk7p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:51:37.133203 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:37.133180    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e860760-2ab7-4db2-80af-91def4cdc1dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e860760-2ab7-4db2-80af-91def4cdc1dd" (UID: "5e860760-2ab7-4db2-80af-91def4cdc1dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:51:37.209538 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:37.209519    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e860760-2ab7-4db2-80af-91def4cdc1dd-model-cache\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:51:37.209538 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:37.209539    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk7p2\" (UniqueName: \"kubernetes.io/projected/5e860760-2ab7-4db2-80af-91def4cdc1dd-kube-api-access-fk7p2\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:51:37.971322 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:37.971291    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mq9nl"
Sep 17 16:51:39.309332 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:39.309303    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:51:39.409991 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:39.409964    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:51:50.955750 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:50.955712    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:51:51.011765 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.011741    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"]
Sep 17 16:51:51.012046 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.012034    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e860760-2ab7-4db2-80af-91def4cdc1dd" containerName="download-hf-model"
Sep 17 16:51:51.012102 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.012047    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e860760-2ab7-4db2-80af-91def4cdc1dd" containerName="download-hf-model"
Sep 17 16:51:51.012102 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.012062    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e860760-2ab7-4db2-80af-91def4cdc1dd" containerName="s3-init"
Sep 17 16:51:51.012102 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.012067    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e860760-2ab7-4db2-80af-91def4cdc1dd" containerName="s3-init"
Sep 17 16:51:51.012199 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.012121    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e860760-2ab7-4db2-80af-91def4cdc1dd" containerName="s3-init"
Sep 17 16:51:51.014138 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.014124    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.016804 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.016777    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 16:51:51.016804 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.016808    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 16:51:51.017054 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.016817    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:51:51.017054 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.016828    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:51:51.017054 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.016857    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlgx8\""
Sep 17 16:51:51.017054 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.016983    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:51:51.017233 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.017183    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:51:51.025876 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.025856    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"]
Sep 17 16:51:51.091112 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.091086    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:51:51.190207 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.190181    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:51:51.190973 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.190951    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"]
Sep 17 16:51:51.193664 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.193645    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.195884 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.195863    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 16:51:51.203317 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.203292    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"]
Sep 17 16:51:51.208581 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.208534    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.208662 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.208585    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nkpb\" (UniqueName: \"kubernetes.io/projected/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kube-api-access-2nkpb\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.208700 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.208689    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.208798 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.208776    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.208844 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.208812    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.253367 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.253347    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:51:51.278796 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.278763    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"]
Sep 17 16:51:51.281277 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.281259    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.283741 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.283718    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-5ef83-kube-rbac-proxy-sar-config\""
Sep 17 16:51:51.283841 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.283740    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-5ef83-predictor-serving-cert\""
Sep 17 16:51:51.294631 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.294613    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"]
Sep 17 16:51:51.310141 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.310213 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310148    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.310213 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310167    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kjn4\" (UniqueName: \"kubernetes.io/projected/bbbfc53f-4bff-447c-8027-a0daa1992779-kube-api-access-8kjn4\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-gbf7p\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.310213 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310184    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.310363 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310225    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50258a5a-f246-47e2-b63c-580c349c0d7f-proxy-tls\") pod \"isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.310363 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310251    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p6dv\" (UniqueName: \"kubernetes.io/projected/50258a5a-f246-47e2-b63c-580c349c0d7f-kube-api-access-6p6dv\") pod \"isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.310363 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310280    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbbfc53f-4bff-447c-8027-a0daa1992779-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-gbf7p\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.310363 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310312    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbbfc53f-4bff-447c-8027-a0daa1992779-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-gbf7p\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.310363 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310342    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.310555 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50258a5a-f246-47e2-b63c-580c349c0d7f-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.310555 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbbfc53f-4bff-447c-8027-a0daa1992779-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-gbf7p\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.310555 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:51:51.310431    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 16:51:51.310555 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310454    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbbfc53f-4bff-447c-8027-a0daa1992779-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-gbf7p\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.310555 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310487    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.310555 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:51:51.310511    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-proxy-tls podName:72aa0650-68bd-4bfb-aca6-39b48f6ad6ef nodeName:}" failed. No retries permitted until 2026-09-17 16:51:51.810490157 +0000 UTC m=+577.376601869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-proxy-tls") pod "isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" (UID: "72aa0650-68bd-4bfb-aca6-39b48f6ad6ef") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 16:51:51.310555 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310547    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nkpb\" (UniqueName: \"kubernetes.io/projected/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kube-api-access-2nkpb\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.310892 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310582    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50258a5a-f246-47e2-b63c-580c349c0d7f-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.310892 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50258a5a-f246-47e2-b63c-580c349c0d7f-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.310892 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310804    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.310892 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.310823    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.322264 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.322243    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nkpb\" (UniqueName: \"kubernetes.io/projected/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kube-api-access-2nkpb\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.411700 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.411675    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50258a5a-f246-47e2-b63c-580c349c0d7f-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.411803 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.411733    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kjn4\" (UniqueName: \"kubernetes.io/projected/bbbfc53f-4bff-447c-8027-a0daa1992779-kube-api-access-8kjn4\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-gbf7p\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.411803 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.411778    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50258a5a-f246-47e2-b63c-580c349c0d7f-proxy-tls\") pod \"isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.411950 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:51:51.411870    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-5ef83-predictor-serving-cert" not found
Sep 17 16:51:51.411950 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.411891    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p6dv\" (UniqueName: \"kubernetes.io/projected/50258a5a-f246-47e2-b63c-580c349c0d7f-kube-api-access-6p6dv\") pod \"isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.412083 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:51:51.411966    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50258a5a-f246-47e2-b63c-580c349c0d7f-proxy-tls podName:50258a5a-f246-47e2-b63c-580c349c0d7f nodeName:}" failed. No retries permitted until 2026-09-17 16:51:51.911946714 +0000 UTC m=+577.478058415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/50258a5a-f246-47e2-b63c-580c349c0d7f-proxy-tls") pod "isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" (UID: "50258a5a-f246-47e2-b63c-580c349c0d7f") : secret "isvc-raw-sklearn-batcher-5ef83-predictor-serving-cert" not found
Sep 17 16:51:51.412083 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.411998    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbbfc53f-4bff-447c-8027-a0daa1992779-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-gbf7p\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.412083 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.412034    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbbfc53f-4bff-447c-8027-a0daa1992779-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-gbf7p\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.412263 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.412111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50258a5a-f246-47e2-b63c-580c349c0d7f-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.412263 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.412149    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbbfc53f-4bff-447c-8027-a0daa1992779-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-gbf7p\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.412263 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.412180    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbbfc53f-4bff-447c-8027-a0daa1992779-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-gbf7p\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.412263 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.412220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50258a5a-f246-47e2-b63c-580c349c0d7f-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.412457 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.412393    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbbfc53f-4bff-447c-8027-a0daa1992779-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-gbf7p\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.412457 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.412423    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50258a5a-f246-47e2-b63c-580c349c0d7f-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.412569 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.412475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50258a5a-f246-47e2-b63c-580c349c0d7f-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.412750 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.412730    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbbfc53f-4bff-447c-8027-a0daa1992779-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-gbf7p\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.412808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.412753    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50258a5a-f246-47e2-b63c-580c349c0d7f-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.412808 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.412788    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbbfc53f-4bff-447c-8027-a0daa1992779-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-gbf7p\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.414594 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.414569    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbbfc53f-4bff-447c-8027-a0daa1992779-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-gbf7p\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.422834 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.422814    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kjn4\" (UniqueName: \"kubernetes.io/projected/bbbfc53f-4bff-447c-8027-a0daa1992779-kube-api-access-8kjn4\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-gbf7p\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.423198 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.423182    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p6dv\" (UniqueName: \"kubernetes.io/projected/50258a5a-f246-47e2-b63c-580c349c0d7f-kube-api-access-6p6dv\") pod \"isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.455554 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.455534    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:51:51.503472 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.503425    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:51:51.557054 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.557028    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:51:51.770993 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.770971    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"]
Sep 17 16:51:51.772982 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:51:51.772955    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbfc53f_4bff_447c_8027_a0daa1992779.slice/crio-afd96231e4d3abf2f00b268cd0407be8836d3ebc56cf6802391f372532b371c5 WatchSource:0}: Error finding container afd96231e4d3abf2f00b268cd0407be8836d3ebc56cf6802391f372532b371c5: Status 404 returned error can't find the container with id afd96231e4d3abf2f00b268cd0407be8836d3ebc56cf6802391f372532b371c5
Sep 17 16:51:51.816734 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.816711    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.818813 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.818794    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:51.917896 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.917868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50258a5a-f246-47e2-b63c-580c349c0d7f-proxy-tls\") pod \"isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.920036 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.920016    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50258a5a-f246-47e2-b63c-580c349c0d7f-proxy-tls\") pod \"isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:51.925511 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:51.925495    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:51:52.024665 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:52.024600    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" event={"ID":"bbbfc53f-4bff-447c-8027-a0daa1992779","Type":"ContainerStarted","Data":"37c721a5994ca7091b48ea025735905f7b10cf7e512fd7581dbae9da4d755bed"}
Sep 17 16:51:52.024990 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:52.024671    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" event={"ID":"bbbfc53f-4bff-447c-8027-a0daa1992779","Type":"ContainerStarted","Data":"afd96231e4d3abf2f00b268cd0407be8836d3ebc56cf6802391f372532b371c5"}
Sep 17 16:51:52.191303 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:52.191267    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:51:52.197418 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:52.197382    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"]
Sep 17 16:51:52.201319 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:51:52.201293    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72aa0650_68bd_4bfb_aca6_39b48f6ad6ef.slice/crio-367feb136450e2447d5a7a124aa1f3b98d46db012f787467e42bd0155d54b15a WatchSource:0}: Error finding container 367feb136450e2447d5a7a124aa1f3b98d46db012f787467e42bd0155d54b15a: Status 404 returned error can't find the container with id 367feb136450e2447d5a7a124aa1f3b98d46db012f787467e42bd0155d54b15a
Sep 17 16:51:52.472240 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:52.472207    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"]
Sep 17 16:51:52.475821 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:51:52.475778    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50258a5a_f246_47e2_b63c_580c349c0d7f.slice/crio-2600c8041114b0c92c08677741944bd8fccdfe27ca43bc870f77c8c65e4c047d WatchSource:0}: Error finding container 2600c8041114b0c92c08677741944bd8fccdfe27ca43bc870f77c8c65e4c047d: Status 404 returned error can't find the container with id 2600c8041114b0c92c08677741944bd8fccdfe27ca43bc870f77c8c65e4c047d
Sep 17 16:51:53.031134 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:53.031099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" event={"ID":"50258a5a-f246-47e2-b63c-580c349c0d7f","Type":"ContainerStarted","Data":"11d31e39e3baa2e2544d7f640cd81dd313c22f78a617ec84b0c4c1306f903b0c"}
Sep 17 16:51:53.031560 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:53.031185    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" event={"ID":"50258a5a-f246-47e2-b63c-580c349c0d7f","Type":"ContainerStarted","Data":"2600c8041114b0c92c08677741944bd8fccdfe27ca43bc870f77c8c65e4c047d"}
Sep 17 16:51:53.032705 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:53.032675    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" event={"ID":"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef","Type":"ContainerStarted","Data":"407e94f084b91098af687d167717c7420ea0777e92dce7e449195c22d4510d09"}
Sep 17 16:51:53.032835 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:53.032728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" event={"ID":"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef","Type":"ContainerStarted","Data":"367feb136450e2447d5a7a124aa1f3b98d46db012f787467e42bd0155d54b15a"}
Sep 17 16:51:56.045621 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:56.045596    2565 generic.go:356] "Generic (PLEG): container finished" podID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerID="407e94f084b91098af687d167717c7420ea0777e92dce7e449195c22d4510d09" exitCode=0
Sep 17 16:51:56.045978 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:56.045669    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" event={"ID":"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef","Type":"ContainerDied","Data":"407e94f084b91098af687d167717c7420ea0777e92dce7e449195c22d4510d09"}
Sep 17 16:51:56.047374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:56.047285    2565 generic.go:356] "Generic (PLEG): container finished" podID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerID="37c721a5994ca7091b48ea025735905f7b10cf7e512fd7581dbae9da4d755bed" exitCode=0
Sep 17 16:51:56.047374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:56.047330    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" event={"ID":"bbbfc53f-4bff-447c-8027-a0daa1992779","Type":"ContainerDied","Data":"37c721a5994ca7091b48ea025735905f7b10cf7e512fd7581dbae9da4d755bed"}
Sep 17 16:51:57.084756 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:57.084667    2565 generic.go:356] "Generic (PLEG): container finished" podID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerID="11d31e39e3baa2e2544d7f640cd81dd313c22f78a617ec84b0c4c1306f903b0c" exitCode=0
Sep 17 16:51:57.085282 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:51:57.084724    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" event={"ID":"50258a5a-f246-47e2-b63c-580c349c0d7f","Type":"ContainerDied","Data":"11d31e39e3baa2e2544d7f640cd81dd313c22f78a617ec84b0c4c1306f903b0c"}
Sep 17 16:52:18.264815 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:18.264775    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" event={"ID":"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef","Type":"ContainerStarted","Data":"ff1bb0ee8ec818f387dbb30bcc9d3f647496b985baf4b2a542081a3f4b354894"}
Sep 17 16:52:18.270789 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:18.270757    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" event={"ID":"bbbfc53f-4bff-447c-8027-a0daa1992779","Type":"ContainerStarted","Data":"11e6ea3639d16caca58c7e13523e76bd2953f6602c94d6ce01caa339b266d6bf"}
Sep 17 16:52:18.274780 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:18.274702    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" event={"ID":"50258a5a-f246-47e2-b63c-580c349c0d7f","Type":"ContainerStarted","Data":"727cd66383f22804614edcff865d978c63842a0307e4e6f6419db117de257021"}
Sep 17 16:52:21.292592 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:21.292552    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" event={"ID":"50258a5a-f246-47e2-b63c-580c349c0d7f","Type":"ContainerStarted","Data":"5eb982c83a712915b01b3d02db3c79f1f343a79e27685ca278e962393e0fefea"}
Sep 17 16:52:21.295337 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:21.295267    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" event={"ID":"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef","Type":"ContainerStarted","Data":"3ccedb8181b91d2882f60a31fdcfad5ea3c9878478236e2687b7a16b32346290"}
Sep 17 16:52:21.295482 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:21.295422    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:52:21.295604 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:21.295584    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:52:21.296830 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:21.296795    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 16:52:21.298056 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:21.298032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" event={"ID":"bbbfc53f-4bff-447c-8027-a0daa1992779","Type":"ContainerStarted","Data":"f46c0a6332515ecbaf0168c983f34c19a70063776391e1e9826b9605579d369c"}
Sep 17 16:52:21.298330 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:21.298305    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:52:21.298429 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:21.298355    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:52:21.299836 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:21.299808    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 16:52:21.314251 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:21.314204    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" podStartSLOduration=3.351012775 podStartE2EDuration="31.314187659s" totalImagesPullingTime="24.212476907s" totalInitContainerRuntime="3.750697977s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:51:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:52:21.312046595 +0000 UTC m=+606.878158321" watchObservedRunningTime="2026-09-17 16:52:21.314187659 +0000 UTC m=+606.880299383"
Sep 17 16:52:21.331761 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:21.331726    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" podStartSLOduration=1.9593026660000001 podStartE2EDuration="30.331716517s" totalImagesPullingTime="24.392480313s" totalInitContainerRuntime="3.979933538s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:51:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:52:21.330860932 +0000 UTC m=+606.896972654" watchObservedRunningTime="2026-09-17 16:52:21.331716517 +0000 UTC m=+606.897828238"
Sep 17 16:52:22.302443 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:22.302401    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 16:52:22.302870 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:22.302408    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 16:52:23.308507 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:23.308470    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" event={"ID":"50258a5a-f246-47e2-b63c-580c349c0d7f","Type":"ContainerStarted","Data":"9c9f29a2631af7210a0a4c29564979a8af688dfd453a296619aa97306d1292ac"}
Sep 17 16:52:23.308840 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:23.308560    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:52:23.332678 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:23.332633    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podStartSLOduration=2.505446223 podStartE2EDuration="32.332618496s" totalImagesPullingTime="25.83703676s" totalInitContainerRuntime="3.990135513s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:51:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:52:23.332375228 +0000 UTC m=+608.898486963" watchObservedRunningTime="2026-09-17 16:52:23.332618496 +0000 UTC m=+608.898730218"
Sep 17 16:52:24.313325 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:24.313294    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:52:24.313689 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:24.313364    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:52:24.314701 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:24.314670    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 16:52:24.315390 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:24.315361    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:52:25.317825 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:25.317785    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 16:52:25.318238 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:25.318097    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:52:25.321574 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:25.321555    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:52:26.321314 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:26.321274    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 16:52:26.321699 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:26.321614    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:52:27.306641 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:27.306609    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:52:27.307024 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:27.307002    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:52:27.307139 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:27.307109    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 16:52:27.307561 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:27.307539    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 16:52:36.322212 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:36.322120    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 16:52:36.322644 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:36.322617    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:52:37.307572 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:37.307534    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 16:52:37.307739 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:37.307534    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 16:52:46.321660 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:46.321611    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 16:52:46.322082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:46.322016    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:52:47.307207 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:47.307170    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 16:52:47.307458 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:47.307434    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 16:52:56.321249 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:56.321193    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 16:52:56.321770 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:56.321741    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:52:57.307175 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:57.307130    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 16:52:57.307542 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:52:57.307510    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 16:53:06.321651 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:06.321606    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 16:53:06.322172 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:06.322146    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:53:07.307339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:07.307294    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 16:53:07.307572 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:07.307416    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 16:53:09.067184 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:09.067150    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:53:16.321448 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:16.321397    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 16:53:16.321930 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:16.321884    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:53:17.307104 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:17.307067    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 16:53:17.307520 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:17.307498    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 16:53:26.322184 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:26.322133    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 16:53:26.322660 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:26.322626    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:53:27.307356 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:27.307319    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 17 16:53:27.307545 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:27.307455    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 17 16:53:36.322132 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:36.322092    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:53:36.322499 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:36.322309    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:53:37.307646 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:37.307621    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:53:37.308163 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:37.308141    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:53:37.433553 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:37.433524    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"]
Sep 17 16:53:37.617821 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:37.617759    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kserve-container" containerID="cri-o://11e6ea3639d16caca58c7e13523e76bd2953f6602c94d6ce01caa339b266d6bf" gracePeriod=30
Sep 17 16:53:37.617998 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:37.617786    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kube-rbac-proxy" containerID="cri-o://f46c0a6332515ecbaf0168c983f34c19a70063776391e1e9826b9605579d369c" gracePeriod=30
Sep 17 16:53:38.623867 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:38.623832    2565 generic.go:356] "Generic (PLEG): container finished" podID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerID="f46c0a6332515ecbaf0168c983f34c19a70063776391e1e9826b9605579d369c" exitCode=2
Sep 17 16:53:38.624316 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:38.623902    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" event={"ID":"bbbfc53f-4bff-447c-8027-a0daa1992779","Type":"ContainerDied","Data":"f46c0a6332515ecbaf0168c983f34c19a70063776391e1e9826b9605579d369c"}
Sep 17 16:53:40.903235 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:40.903205    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"]
Sep 17 16:53:40.903687 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:40.903423    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kserve-container" containerID="cri-o://ff1bb0ee8ec818f387dbb30bcc9d3f647496b985baf4b2a542081a3f4b354894" gracePeriod=30
Sep 17 16:53:40.903687 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:40.903452    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kube-rbac-proxy" containerID="cri-o://3ccedb8181b91d2882f60a31fdcfad5ea3c9878478236e2687b7a16b32346290" gracePeriod=30
Sep 17 16:53:40.930964 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:40.930938    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:53:40.972715 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:40.972692    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"]
Sep 17 16:53:40.976120 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:40.976100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:40.984721 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:40.984701    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"]
Sep 17 16:53:41.076593 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.076572    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fghkv\" (UniqueName: \"kubernetes.io/projected/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kube-api-access-fghkv\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-rwk9c\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:41.076686 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.076602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-rwk9c\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:41.076686 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.076646    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-rwk9c\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:41.076686 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.076674    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-rwk9c\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:41.076789 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.076716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-rwk9c\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:41.177212 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.177155    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fghkv\" (UniqueName: \"kubernetes.io/projected/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kube-api-access-fghkv\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-rwk9c\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:41.177212 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.177188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-rwk9c\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:41.177332 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.177310    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-rwk9c\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:41.177372 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.177363    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-rwk9c\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:41.177419 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.177407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-rwk9c\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:41.177683 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.177663    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-rwk9c\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:41.178017 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.177996    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-rwk9c\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:41.178113 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.178091    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-rwk9c\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:41.179443 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.179423    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-proxy-tls\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-rwk9c\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:41.185207 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.185186    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fghkv\" (UniqueName: \"kubernetes.io/projected/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kube-api-access-fghkv\") pod \"isvc-canary-rollback-predictor-696c5cc6cc-rwk9c\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:41.286315 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.286290    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:41.554339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.554302    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"]
Sep 17 16:53:41.608771 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:53:41.608744    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf6921d_2cbf_4f63_b6b0_3a24c67114d7.slice/crio-a067b2bc9a485c6190b2eb96c0f563caef338c8c25efd4696db9873d5ac5d736 WatchSource:0}: Error finding container a067b2bc9a485c6190b2eb96c0f563caef338c8c25efd4696db9873d5ac5d736: Status 404 returned error can't find the container with id a067b2bc9a485c6190b2eb96c0f563caef338c8c25efd4696db9873d5ac5d736
Sep 17 16:53:41.637940 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.637839    2565 generic.go:356] "Generic (PLEG): container finished" podID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerID="3ccedb8181b91d2882f60a31fdcfad5ea3c9878478236e2687b7a16b32346290" exitCode=2
Sep 17 16:53:41.637940 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.637908    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" event={"ID":"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef","Type":"ContainerDied","Data":"3ccedb8181b91d2882f60a31fdcfad5ea3c9878478236e2687b7a16b32346290"}
Sep 17 16:53:41.640976 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.640950    2565 generic.go:356] "Generic (PLEG): container finished" podID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerID="11e6ea3639d16caca58c7e13523e76bd2953f6602c94d6ce01caa339b266d6bf" exitCode=0
Sep 17 16:53:41.641089 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.641032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" event={"ID":"bbbfc53f-4bff-447c-8027-a0daa1992779","Type":"ContainerDied","Data":"11e6ea3639d16caca58c7e13523e76bd2953f6602c94d6ce01caa339b266d6bf"}
Sep 17 16:53:41.642479 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.642459    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" event={"ID":"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7","Type":"ContainerStarted","Data":"a067b2bc9a485c6190b2eb96c0f563caef338c8c25efd4696db9873d5ac5d736"}
Sep 17 16:53:41.769255 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.769232    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:53:41.881902 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.881875    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbbfc53f-4bff-447c-8027-a0daa1992779-cabundle-cert\") pod \"bbbfc53f-4bff-447c-8027-a0daa1992779\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") "
Sep 17 16:53:41.882084 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.881977    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbbfc53f-4bff-447c-8027-a0daa1992779-kserve-provision-location\") pod \"bbbfc53f-4bff-447c-8027-a0daa1992779\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") "
Sep 17 16:53:41.882084 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.882013    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbbfc53f-4bff-447c-8027-a0daa1992779-proxy-tls\") pod \"bbbfc53f-4bff-447c-8027-a0daa1992779\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") "
Sep 17 16:53:41.882084 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.882032    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kjn4\" (UniqueName: \"kubernetes.io/projected/bbbfc53f-4bff-447c-8027-a0daa1992779-kube-api-access-8kjn4\") pod \"bbbfc53f-4bff-447c-8027-a0daa1992779\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") "
Sep 17 16:53:41.882084 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.882057    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbbfc53f-4bff-447c-8027-a0daa1992779-kube-rbac-proxy-sar-config\") pod \"bbbfc53f-4bff-447c-8027-a0daa1992779\" (UID: \"bbbfc53f-4bff-447c-8027-a0daa1992779\") "
Sep 17 16:53:41.882303 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.882281    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbfc53f-4bff-447c-8027-a0daa1992779-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bbbfc53f-4bff-447c-8027-a0daa1992779" (UID: "bbbfc53f-4bff-447c-8027-a0daa1992779"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:53:41.882360 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.882320    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbfc53f-4bff-447c-8027-a0daa1992779-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bbbfc53f-4bff-447c-8027-a0daa1992779" (UID: "bbbfc53f-4bff-447c-8027-a0daa1992779"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:53:41.882723 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.882698    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbfc53f-4bff-447c-8027-a0daa1992779-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bbbfc53f-4bff-447c-8027-a0daa1992779" (UID: "bbbfc53f-4bff-447c-8027-a0daa1992779"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:53:41.884280 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.884258    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbfc53f-4bff-447c-8027-a0daa1992779-kube-api-access-8kjn4" (OuterVolumeSpecName: "kube-api-access-8kjn4") pod "bbbfc53f-4bff-447c-8027-a0daa1992779" (UID: "bbbfc53f-4bff-447c-8027-a0daa1992779"). InnerVolumeSpecName "kube-api-access-8kjn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:53:41.884528 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.884512    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbbfc53f-4bff-447c-8027-a0daa1992779-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bbbfc53f-4bff-447c-8027-a0daa1992779" (UID: "bbbfc53f-4bff-447c-8027-a0daa1992779"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:53:41.982830 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.982805    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbbfc53f-4bff-447c-8027-a0daa1992779-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:53:41.982830 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.982828    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kjn4\" (UniqueName: \"kubernetes.io/projected/bbbfc53f-4bff-447c-8027-a0daa1992779-kube-api-access-8kjn4\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:53:41.983193 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.982839    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbbfc53f-4bff-447c-8027-a0daa1992779-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:53:41.983193 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.982848    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbbfc53f-4bff-447c-8027-a0daa1992779-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:53:41.983193 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:41.982856    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbbfc53f-4bff-447c-8027-a0daa1992779-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:53:42.303120 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:42.303034    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Sep 17 16:53:42.649051 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:42.649024    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"
Sep 17 16:53:42.649235 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:42.649030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p" event={"ID":"bbbfc53f-4bff-447c-8027-a0daa1992779","Type":"ContainerDied","Data":"afd96231e4d3abf2f00b268cd0407be8836d3ebc56cf6802391f372532b371c5"}
Sep 17 16:53:42.649235 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:42.649219    2565 scope.go:117] "RemoveContainer" containerID="f46c0a6332515ecbaf0168c983f34c19a70063776391e1e9826b9605579d369c"
Sep 17 16:53:42.650737 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:42.650713    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" event={"ID":"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7","Type":"ContainerStarted","Data":"f3cf1b05fc489cdf2387aad168b30dca887343307afe65002656f049a6478d64"}
Sep 17 16:53:42.659553 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:42.659536    2565 scope.go:117] "RemoveContainer" containerID="11e6ea3639d16caca58c7e13523e76bd2953f6602c94d6ce01caa339b266d6bf"
Sep 17 16:53:42.666655 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:42.666638    2565 scope.go:117] "RemoveContainer" containerID="37c721a5994ca7091b48ea025735905f7b10cf7e512fd7581dbae9da4d755bed"
Sep 17 16:53:42.685179 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:42.685152    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"]
Sep 17 16:53:42.688952 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:42.688933    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-gbf7p"]
Sep 17 16:53:42.823793 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:42.823771    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" path="/var/lib/kubelet/pods/bbbfc53f-4bff-447c-8027-a0daa1992779/volumes"
Sep 17 16:53:45.038767 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.038742    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:53:45.203754 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.203725    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kserve-provision-location\") pod \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") "
Sep 17 16:53:45.203899 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.203790    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nkpb\" (UniqueName: \"kubernetes.io/projected/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kube-api-access-2nkpb\") pod \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") "
Sep 17 16:53:45.203899 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.203828    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kube-rbac-proxy-sar-config\") pod \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") "
Sep 17 16:53:45.203899 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.203864    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-proxy-tls\") pod \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") "
Sep 17 16:53:45.204082 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.203935    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-cabundle-cert\") pod \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\" (UID: \"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef\") "
Sep 17 16:53:45.204141 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.204082    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" (UID: "72aa0650-68bd-4bfb-aca6-39b48f6ad6ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:53:45.204199 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.204173    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:53:45.204199 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.204177    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" (UID: "72aa0650-68bd-4bfb-aca6-39b48f6ad6ef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:53:45.204344 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.204325    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" (UID: "72aa0650-68bd-4bfb-aca6-39b48f6ad6ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:53:45.205859 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.205834    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kube-api-access-2nkpb" (OuterVolumeSpecName: "kube-api-access-2nkpb") pod "72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" (UID: "72aa0650-68bd-4bfb-aca6-39b48f6ad6ef"). InnerVolumeSpecName "kube-api-access-2nkpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:53:45.205992 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.205874    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" (UID: "72aa0650-68bd-4bfb-aca6-39b48f6ad6ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:53:45.305326 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.305305    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:53:45.305326 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.305324    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nkpb\" (UniqueName: \"kubernetes.io/projected/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kube-api-access-2nkpb\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:53:45.305453 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.305334    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:53:45.305453 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.305344    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:53:45.666770 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.666738    2565 generic.go:356] "Generic (PLEG): container finished" podID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerID="f3cf1b05fc489cdf2387aad168b30dca887343307afe65002656f049a6478d64" exitCode=0
Sep 17 16:53:45.666936 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.666819    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" event={"ID":"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7","Type":"ContainerDied","Data":"f3cf1b05fc489cdf2387aad168b30dca887343307afe65002656f049a6478d64"}
Sep 17 16:53:45.668930 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.668893    2565 generic.go:356] "Generic (PLEG): container finished" podID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerID="ff1bb0ee8ec818f387dbb30bcc9d3f647496b985baf4b2a542081a3f4b354894" exitCode=0
Sep 17 16:53:45.669033 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.668953    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" event={"ID":"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef","Type":"ContainerDied","Data":"ff1bb0ee8ec818f387dbb30bcc9d3f647496b985baf4b2a542081a3f4b354894"}
Sep 17 16:53:45.669033 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.669010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz" event={"ID":"72aa0650-68bd-4bfb-aca6-39b48f6ad6ef","Type":"ContainerDied","Data":"367feb136450e2447d5a7a124aa1f3b98d46db012f787467e42bd0155d54b15a"}
Sep 17 16:53:45.669110 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.669037    2565 scope.go:117] "RemoveContainer" containerID="3ccedb8181b91d2882f60a31fdcfad5ea3c9878478236e2687b7a16b32346290"
Sep 17 16:53:45.669110 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.668962    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"
Sep 17 16:53:45.680074 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.680048    2565 scope.go:117] "RemoveContainer" containerID="ff1bb0ee8ec818f387dbb30bcc9d3f647496b985baf4b2a542081a3f4b354894"
Sep 17 16:53:45.689406 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.689388    2565 scope.go:117] "RemoveContainer" containerID="407e94f084b91098af687d167717c7420ea0777e92dce7e449195c22d4510d09"
Sep 17 16:53:45.696624 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.696603    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"]
Sep 17 16:53:45.700645 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.700624    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7ffd87447b-wtgjz"]
Sep 17 16:53:45.704476 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.704460    2565 scope.go:117] "RemoveContainer" containerID="3ccedb8181b91d2882f60a31fdcfad5ea3c9878478236e2687b7a16b32346290"
Sep 17 16:53:45.704866 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:53:45.704762    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ccedb8181b91d2882f60a31fdcfad5ea3c9878478236e2687b7a16b32346290\": container with ID starting with 3ccedb8181b91d2882f60a31fdcfad5ea3c9878478236e2687b7a16b32346290 not found: ID does not exist" containerID="3ccedb8181b91d2882f60a31fdcfad5ea3c9878478236e2687b7a16b32346290"
Sep 17 16:53:45.704866 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.704792    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ccedb8181b91d2882f60a31fdcfad5ea3c9878478236e2687b7a16b32346290"} err="failed to get container status \"3ccedb8181b91d2882f60a31fdcfad5ea3c9878478236e2687b7a16b32346290\": rpc error: code = NotFound desc = could not find container \"3ccedb8181b91d2882f60a31fdcfad5ea3c9878478236e2687b7a16b32346290\": container with ID starting with 3ccedb8181b91d2882f60a31fdcfad5ea3c9878478236e2687b7a16b32346290 not found: ID does not exist"
Sep 17 16:53:45.704866 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.704812    2565 scope.go:117] "RemoveContainer" containerID="ff1bb0ee8ec818f387dbb30bcc9d3f647496b985baf4b2a542081a3f4b354894"
Sep 17 16:53:45.705138 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:53:45.705119    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1bb0ee8ec818f387dbb30bcc9d3f647496b985baf4b2a542081a3f4b354894\": container with ID starting with ff1bb0ee8ec818f387dbb30bcc9d3f647496b985baf4b2a542081a3f4b354894 not found: ID does not exist" containerID="ff1bb0ee8ec818f387dbb30bcc9d3f647496b985baf4b2a542081a3f4b354894"
Sep 17 16:53:45.705258 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.705148    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1bb0ee8ec818f387dbb30bcc9d3f647496b985baf4b2a542081a3f4b354894"} err="failed to get container status \"ff1bb0ee8ec818f387dbb30bcc9d3f647496b985baf4b2a542081a3f4b354894\": rpc error: code = NotFound desc = could not find container \"ff1bb0ee8ec818f387dbb30bcc9d3f647496b985baf4b2a542081a3f4b354894\": container with ID starting with ff1bb0ee8ec818f387dbb30bcc9d3f647496b985baf4b2a542081a3f4b354894 not found: ID does not exist"
Sep 17 16:53:45.705258 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.705170    2565 scope.go:117] "RemoveContainer" containerID="407e94f084b91098af687d167717c7420ea0777e92dce7e449195c22d4510d09"
Sep 17 16:53:45.705442 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:53:45.705424    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407e94f084b91098af687d167717c7420ea0777e92dce7e449195c22d4510d09\": container with ID starting with 407e94f084b91098af687d167717c7420ea0777e92dce7e449195c22d4510d09 not found: ID does not exist" containerID="407e94f084b91098af687d167717c7420ea0777e92dce7e449195c22d4510d09"
Sep 17 16:53:45.705481 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.705449    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407e94f084b91098af687d167717c7420ea0777e92dce7e449195c22d4510d09"} err="failed to get container status \"407e94f084b91098af687d167717c7420ea0777e92dce7e449195c22d4510d09\": rpc error: code = NotFound desc = could not find container \"407e94f084b91098af687d167717c7420ea0777e92dce7e449195c22d4510d09\": container with ID starting with 407e94f084b91098af687d167717c7420ea0777e92dce7e449195c22d4510d09 not found: ID does not exist"
Sep 17 16:53:45.948470 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.948390    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"]
Sep 17 16:53:45.948850 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.948697    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container" containerID="cri-o://727cd66383f22804614edcff865d978c63842a0307e4e6f6419db117de257021" gracePeriod=30
Sep 17 16:53:45.948850 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.948716    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent" containerID="cri-o://9c9f29a2631af7210a0a4c29564979a8af688dfd453a296619aa97306d1292ac" gracePeriod=30
Sep 17 16:53:45.948850 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.948741    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kube-rbac-proxy" containerID="cri-o://5eb982c83a712915b01b3d02db3c79f1f343a79e27685ca278e962393e0fefea" gracePeriod=30
Sep 17 16:53:45.974432 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:45.974408    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:53:46.070209 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.070184    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:53:46.091085 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091059    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"]
Sep 17 16:53:46.091376 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091363    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kserve-container"
Sep 17 16:53:46.091426 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091376    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kserve-container"
Sep 17 16:53:46.091426 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091388    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kserve-container"
Sep 17 16:53:46.091426 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091393    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kserve-container"
Sep 17 16:53:46.091426 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091403    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="storage-initializer"
Sep 17 16:53:46.091426 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091408    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="storage-initializer"
Sep 17 16:53:46.091426 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091417    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kube-rbac-proxy"
Sep 17 16:53:46.091426 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091422    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kube-rbac-proxy"
Sep 17 16:53:46.091657 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091436    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kube-rbac-proxy"
Sep 17 16:53:46.091657 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091441    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kube-rbac-proxy"
Sep 17 16:53:46.091657 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091449    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="storage-initializer"
Sep 17 16:53:46.091657 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091454    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="storage-initializer"
Sep 17 16:53:46.091657 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091501    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kserve-container"
Sep 17 16:53:46.091657 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091509    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kube-rbac-proxy"
Sep 17 16:53:46.091657 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091516    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" containerName="kube-rbac-proxy"
Sep 17 16:53:46.091657 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.091524    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbbfc53f-4bff-447c-8027-a0daa1992779" containerName="kserve-container"
Sep 17 16:53:46.094548 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.094532    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.097083 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.097037    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-48e77-kube-rbac-proxy-sar-config\""
Sep 17 16:53:46.097083 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.097076    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-48e77-predictor-serving-cert\""
Sep 17 16:53:46.108424 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.108169    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"]
Sep 17 16:53:46.171810 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.171783    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:53:46.213899 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.213833    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvbb7\" (UniqueName: \"kubernetes.io/projected/71333a7b-dba5-4488-8dfc-930854c8dfe3-kube-api-access-vvbb7\") pod \"isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.213899 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.213865    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71333a7b-dba5-4488-8dfc-930854c8dfe3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.213899 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.213886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71333a7b-dba5-4488-8dfc-930854c8dfe3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.214104 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.213938    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71333a7b-dba5-4488-8dfc-930854c8dfe3-proxy-tls\") pod \"isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.214104 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.213961    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71333a7b-dba5-4488-8dfc-930854c8dfe3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.238313 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.238283    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"]
Sep 17 16:53:46.240990 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.240971    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:53:46.241623 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.241607    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.244161 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.244145    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-48e77-predictor-serving-cert\""
Sep 17 16:53:46.244510 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.244494    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-48e77-kube-rbac-proxy-sar-config\""
Sep 17 16:53:46.253196 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.253175    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"]
Sep 17 16:53:46.315112 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.315090    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvbb7\" (UniqueName: \"kubernetes.io/projected/71333a7b-dba5-4488-8dfc-930854c8dfe3-kube-api-access-vvbb7\") pod \"isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.315220 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.315121    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.315220 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.315141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f52e550-2266-464e-bbc1-b12e1b7f3f55-proxy-tls\") pod \"isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.315220 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.315159    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71333a7b-dba5-4488-8dfc-930854c8dfe3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.315220 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.315191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71333a7b-dba5-4488-8dfc-930854c8dfe3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.315220 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.315214    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.315466 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.315256    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71333a7b-dba5-4488-8dfc-930854c8dfe3-proxy-tls\") pod \"isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.315466 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.315374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phvph\" (UniqueName: \"kubernetes.io/projected/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kube-api-access-phvph\") pod \"isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.315466 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.315406    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71333a7b-dba5-4488-8dfc-930854c8dfe3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.315466 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.315449    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f52e550-2266-464e-bbc1-b12e1b7f3f55-cabundle-cert\") pod \"isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.315668 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.315586    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71333a7b-dba5-4488-8dfc-930854c8dfe3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.315816 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.315796    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71333a7b-dba5-4488-8dfc-930854c8dfe3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.316013 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.315996    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71333a7b-dba5-4488-8dfc-930854c8dfe3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.317546 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.317529    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71333a7b-dba5-4488-8dfc-930854c8dfe3-proxy-tls\") pod \"isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.321219 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.321195    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 16:53:46.321610 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.321588    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:53:46.322954 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.322903    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvbb7\" (UniqueName: \"kubernetes.io/projected/71333a7b-dba5-4488-8dfc-930854c8dfe3-kube-api-access-vvbb7\") pod \"isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.400017 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.399988    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:53:46.404838 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.404822    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:46.415976 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.415956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phvph\" (UniqueName: \"kubernetes.io/projected/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kube-api-access-phvph\") pod \"isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.416079 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.416009    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f52e550-2266-464e-bbc1-b12e1b7f3f55-cabundle-cert\") pod \"isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.416163 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.416135    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.416227 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.416186    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f52e550-2266-464e-bbc1-b12e1b7f3f55-proxy-tls\") pod \"isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.416405 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.416229    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.416635 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.416610    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.416804 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.416787    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f52e550-2266-464e-bbc1-b12e1b7f3f55-cabundle-cert\") pod \"isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.416872 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.416814    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.419311 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.419288    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f52e550-2266-464e-bbc1-b12e1b7f3f55-proxy-tls\") pod \"isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.424251 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.424229    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phvph\" (UniqueName: \"kubernetes.io/projected/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kube-api-access-phvph\") pod \"isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.554421 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.554354    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:53:46.677036 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.676897    2565 generic.go:356] "Generic (PLEG): container finished" podID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerID="5eb982c83a712915b01b3d02db3c79f1f343a79e27685ca278e962393e0fefea" exitCode=2
Sep 17 16:53:46.677036 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.676950    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" event={"ID":"50258a5a-f246-47e2-b63c-580c349c0d7f","Type":"ContainerDied","Data":"5eb982c83a712915b01b3d02db3c79f1f343a79e27685ca278e962393e0fefea"}
Sep 17 16:53:46.678332 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.678306    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"]
Sep 17 16:53:46.679321 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.679300    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" event={"ID":"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7","Type":"ContainerStarted","Data":"e909c5aabf722489f149e767a2359ce72bd6ca11c8f91d2404737d0d2bf89c34"}
Sep 17 16:53:46.679407 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.679343    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" event={"ID":"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7","Type":"ContainerStarted","Data":"8377ed2738f73402a2b6c83911c8b108c0391aeacfaf8ba60c661ffa03dfee49"}
Sep 17 16:53:46.679566 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.679543    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:46.679928 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:53:46.679894    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71333a7b_dba5_4488_8dfc_930854c8dfe3.slice/crio-a06a029fefb3f6db06eb575e105f1eedeb5e653a0a6fdba21f90c43d48d42ef0 WatchSource:0}: Error finding container a06a029fefb3f6db06eb575e105f1eedeb5e653a0a6fdba21f90c43d48d42ef0: Status 404 returned error can't find the container with id a06a029fefb3f6db06eb575e105f1eedeb5e653a0a6fdba21f90c43d48d42ef0
Sep 17 16:53:46.701107 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.701065    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" podStartSLOduration=3.016341203 podStartE2EDuration="6.701052984s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.684711781s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:53:46.698610146 +0000 UTC m=+692.264721885" watchObservedRunningTime="2026-09-17 16:53:46.701052984 +0000 UTC m=+692.267164770"
Sep 17 16:53:46.827072 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:53:46.827043    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f52e550_2266_464e_bbc1_b12e1b7f3f55.slice/crio-fdd7b904f0c2a1a810ac3993d00d11b9bf708b00821892e35e07974998933350 WatchSource:0}: Error finding container fdd7b904f0c2a1a810ac3993d00d11b9bf708b00821892e35e07974998933350: Status 404 returned error can't find the container with id fdd7b904f0c2a1a810ac3993d00d11b9bf708b00821892e35e07974998933350
Sep 17 16:53:46.828868 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.828769    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72aa0650-68bd-4bfb-aca6-39b48f6ad6ef" path="/var/lib/kubelet/pods/72aa0650-68bd-4bfb-aca6-39b48f6ad6ef/volumes"
Sep 17 16:53:46.829261 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:46.829241    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"]
Sep 17 16:53:47.686302 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:47.686268    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" event={"ID":"71333a7b-dba5-4488-8dfc-930854c8dfe3","Type":"ContainerStarted","Data":"757684746c8e2f60dcb4b7f4c8a3086b5741f6427388ea52e5d42bcff117cfab"}
Sep 17 16:53:47.686710 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:47.686343    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" event={"ID":"71333a7b-dba5-4488-8dfc-930854c8dfe3","Type":"ContainerStarted","Data":"a06a029fefb3f6db06eb575e105f1eedeb5e653a0a6fdba21f90c43d48d42ef0"}
Sep 17 16:53:47.688137 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:47.688108    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" event={"ID":"8f52e550-2266-464e-bbc1-b12e1b7f3f55","Type":"ContainerStarted","Data":"9122d24944c370c959a80944f40c7f1bc6c9638166c2a9939f698f41ed96c0b9"}
Sep 17 16:53:47.688275 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:47.688159    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" event={"ID":"8f52e550-2266-464e-bbc1-b12e1b7f3f55","Type":"ContainerStarted","Data":"fdd7b904f0c2a1a810ac3993d00d11b9bf708b00821892e35e07974998933350"}
Sep 17 16:53:47.688442 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:47.688427    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:47.689547 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:47.689518    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 16:53:48.700039 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:48.699998    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 16:53:50.318973 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:50.318876    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 17 16:53:50.711761 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:50.711722    2565 generic.go:356] "Generic (PLEG): container finished" podID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerID="757684746c8e2f60dcb4b7f4c8a3086b5741f6427388ea52e5d42bcff117cfab" exitCode=0
Sep 17 16:53:50.711941 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:50.711804    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" event={"ID":"71333a7b-dba5-4488-8dfc-930854c8dfe3","Type":"ContainerDied","Data":"757684746c8e2f60dcb4b7f4c8a3086b5741f6427388ea52e5d42bcff117cfab"}
Sep 17 16:53:50.714549 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:50.714527    2565 generic.go:356] "Generic (PLEG): container finished" podID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerID="727cd66383f22804614edcff865d978c63842a0307e4e6f6419db117de257021" exitCode=0
Sep 17 16:53:50.714674 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:50.714574    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" event={"ID":"50258a5a-f246-47e2-b63c-580c349c0d7f","Type":"ContainerDied","Data":"727cd66383f22804614edcff865d978c63842a0307e4e6f6419db117de257021"}
Sep 17 16:53:51.723887 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:51.723848    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" event={"ID":"71333a7b-dba5-4488-8dfc-930854c8dfe3","Type":"ContainerStarted","Data":"b38bab56edf049a3d6212be966d0895c29355a78c5f45044276708ef4cf66a75"}
Sep 17 16:53:51.724306 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:51.723943    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" event={"ID":"71333a7b-dba5-4488-8dfc-930854c8dfe3","Type":"ContainerStarted","Data":"f64f8c854f58a2ebbb35c612b600566e38e35a6277de5aa81198c12610352ccf"}
Sep 17 16:53:51.724306 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:51.724172    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:51.725423 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:51.725401    2565 generic.go:356] "Generic (PLEG): container finished" podID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerID="9122d24944c370c959a80944f40c7f1bc6c9638166c2a9939f698f41ed96c0b9" exitCode=0
Sep 17 16:53:51.725509 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:51.725470    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" event={"ID":"8f52e550-2266-464e-bbc1-b12e1b7f3f55","Type":"ContainerDied","Data":"9122d24944c370c959a80944f40c7f1bc6c9638166c2a9939f698f41ed96c0b9"}
Sep 17 16:53:51.726612 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:51.726598    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:53:51.741617 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:51.741574    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" podStartSLOduration=1.937054973 podStartE2EDuration="5.741562052s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.804507079s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:53:51.740300263 +0000 UTC m=+697.306411987" watchObservedRunningTime="2026-09-17 16:53:51.741562052 +0000 UTC m=+697.307673773"
Sep 17 16:53:52.731313 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:52.731282    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:52.732756 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:52.732698    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 16:53:53.705048 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:53.705014    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:53:53.705682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:53.705647    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 16:53:53.737931 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:53.737877    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 16:53:55.318620 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:55.318568    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 17 16:53:56.321636 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:56.321592    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 16:53:56.322112 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:56.321948    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:53:58.743364 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:58.743280    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:53:58.744039 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:53:58.743997    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 16:54:00.318259 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:00.318224    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 17 16:54:00.318602 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:00.318341    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:54:03.705673 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:03.705630    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 16:54:05.318622 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:05.318579    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 17 16:54:06.322066 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:06.322026    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 16:54:06.322518 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:06.322188    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:54:06.322518 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:06.322394    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:54:06.322518 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:06.322514    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:54:08.743854 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:08.743813    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 16:54:10.318819 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:10.318771    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 17 16:54:12.832066 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:12.832026    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" event={"ID":"8f52e550-2266-464e-bbc1-b12e1b7f3f55","Type":"ContainerStarted","Data":"a7af0a521c45cdd317ed3c040c5402b31f4dbf07dbd4eb99997a1f17984ee7f0"}
Sep 17 16:54:12.832534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:12.832099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" event={"ID":"8f52e550-2266-464e-bbc1-b12e1b7f3f55","Type":"ContainerStarted","Data":"ec4f1cefd8417a22739855d925637e75dffda70f1525dc67951929dd24cd8ed9"}
Sep 17 16:54:12.832534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:12.832322    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:54:12.862184 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:12.856891    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" podStartSLOduration=2.650725361 podStartE2EDuration="26.856873409s" totalImagesPullingTime="20.388917594s" totalInitContainerRuntime="3.817230454s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:54:12.854347314 +0000 UTC m=+718.420459037" watchObservedRunningTime="2026-09-17 16:54:12.856873409 +0000 UTC m=+718.422985131"
Sep 17 16:54:13.706591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:13.706556    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 16:54:13.836966 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:13.836939    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:54:13.838159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:13.838132    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 16:54:14.840768 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:14.840728    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 16:54:15.318668 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:15.318626    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 17 16:54:16.321743 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.321676    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 16:54:16.322084 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.321683    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent" probeResult="failure" output="Get \"http://10.134.0.21:9081/\": dial tcp 10.134.0.21:9081: connect: connection refused"
Sep 17 16:54:16.638224 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.638201    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:54:16.686868 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.686845    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50258a5a-f246-47e2-b63c-580c349c0d7f-proxy-tls\") pod \"50258a5a-f246-47e2-b63c-580c349c0d7f\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") "
Sep 17 16:54:16.687000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.686892    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50258a5a-f246-47e2-b63c-580c349c0d7f-cabundle-cert\") pod \"50258a5a-f246-47e2-b63c-580c349c0d7f\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") "
Sep 17 16:54:16.687000 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.686937    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p6dv\" (UniqueName: \"kubernetes.io/projected/50258a5a-f246-47e2-b63c-580c349c0d7f-kube-api-access-6p6dv\") pod \"50258a5a-f246-47e2-b63c-580c349c0d7f\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") "
Sep 17 16:54:16.687086 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.687037    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50258a5a-f246-47e2-b63c-580c349c0d7f-kserve-provision-location\") pod \"50258a5a-f246-47e2-b63c-580c349c0d7f\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") "
Sep 17 16:54:16.687154 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.687137    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50258a5a-f246-47e2-b63c-580c349c0d7f-kube-rbac-proxy-sar-config\") pod \"50258a5a-f246-47e2-b63c-580c349c0d7f\" (UID: \"50258a5a-f246-47e2-b63c-580c349c0d7f\") "
Sep 17 16:54:16.687317 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.687292    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50258a5a-f246-47e2-b63c-580c349c0d7f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "50258a5a-f246-47e2-b63c-580c349c0d7f" (UID: "50258a5a-f246-47e2-b63c-580c349c0d7f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:54:16.687450 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.687314    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50258a5a-f246-47e2-b63c-580c349c0d7f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50258a5a-f246-47e2-b63c-580c349c0d7f" (UID: "50258a5a-f246-47e2-b63c-580c349c0d7f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:54:16.687512 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.687466    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50258a5a-f246-47e2-b63c-580c349c0d7f-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:54:16.687512 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.687486    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50258a5a-f246-47e2-b63c-580c349c0d7f-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:54:16.687512 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.687481    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50258a5a-f246-47e2-b63c-580c349c0d7f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50258a5a-f246-47e2-b63c-580c349c0d7f" (UID: "50258a5a-f246-47e2-b63c-580c349c0d7f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:54:16.688835 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.688816    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50258a5a-f246-47e2-b63c-580c349c0d7f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50258a5a-f246-47e2-b63c-580c349c0d7f" (UID: "50258a5a-f246-47e2-b63c-580c349c0d7f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:54:16.688954 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.688894    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50258a5a-f246-47e2-b63c-580c349c0d7f-kube-api-access-6p6dv" (OuterVolumeSpecName: "kube-api-access-6p6dv") pod "50258a5a-f246-47e2-b63c-580c349c0d7f" (UID: "50258a5a-f246-47e2-b63c-580c349c0d7f"). InnerVolumeSpecName "kube-api-access-6p6dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:54:16.788538 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.788517    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50258a5a-f246-47e2-b63c-580c349c0d7f-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:54:16.788538 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.788540    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p6dv\" (UniqueName: \"kubernetes.io/projected/50258a5a-f246-47e2-b63c-580c349c0d7f-kube-api-access-6p6dv\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:54:16.788657 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.788550    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50258a5a-f246-47e2-b63c-580c349c0d7f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:54:16.852719 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.852650    2565 generic.go:356] "Generic (PLEG): container finished" podID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerID="9c9f29a2631af7210a0a4c29564979a8af688dfd453a296619aa97306d1292ac" exitCode=0
Sep 17 16:54:16.852719 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.852698    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" event={"ID":"50258a5a-f246-47e2-b63c-580c349c0d7f","Type":"ContainerDied","Data":"9c9f29a2631af7210a0a4c29564979a8af688dfd453a296619aa97306d1292ac"}
Sep 17 16:54:16.852873 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.852724    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"
Sep 17 16:54:16.852873 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.852764    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm" event={"ID":"50258a5a-f246-47e2-b63c-580c349c0d7f","Type":"ContainerDied","Data":"2600c8041114b0c92c08677741944bd8fccdfe27ca43bc870f77c8c65e4c047d"}
Sep 17 16:54:16.852873 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.852811    2565 scope.go:117] "RemoveContainer" containerID="9c9f29a2631af7210a0a4c29564979a8af688dfd453a296619aa97306d1292ac"
Sep 17 16:54:16.861128 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.861110    2565 scope.go:117] "RemoveContainer" containerID="5eb982c83a712915b01b3d02db3c79f1f343a79e27685ca278e962393e0fefea"
Sep 17 16:54:16.868386 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.868371    2565 scope.go:117] "RemoveContainer" containerID="727cd66383f22804614edcff865d978c63842a0307e4e6f6419db117de257021"
Sep 17 16:54:16.871412 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.871390    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"]
Sep 17 16:54:16.875386 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.875366    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5ef83-predictor-85f4f6ff9b-7tkkm"]
Sep 17 16:54:16.876348 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.876334    2565 scope.go:117] "RemoveContainer" containerID="11d31e39e3baa2e2544d7f640cd81dd313c22f78a617ec84b0c4c1306f903b0c"
Sep 17 16:54:16.883143 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.883111    2565 scope.go:117] "RemoveContainer" containerID="9c9f29a2631af7210a0a4c29564979a8af688dfd453a296619aa97306d1292ac"
Sep 17 16:54:16.883411 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:54:16.883392    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c9f29a2631af7210a0a4c29564979a8af688dfd453a296619aa97306d1292ac\": container with ID starting with 9c9f29a2631af7210a0a4c29564979a8af688dfd453a296619aa97306d1292ac not found: ID does not exist" containerID="9c9f29a2631af7210a0a4c29564979a8af688dfd453a296619aa97306d1292ac"
Sep 17 16:54:16.883470 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.883418    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c9f29a2631af7210a0a4c29564979a8af688dfd453a296619aa97306d1292ac"} err="failed to get container status \"9c9f29a2631af7210a0a4c29564979a8af688dfd453a296619aa97306d1292ac\": rpc error: code = NotFound desc = could not find container \"9c9f29a2631af7210a0a4c29564979a8af688dfd453a296619aa97306d1292ac\": container with ID starting with 9c9f29a2631af7210a0a4c29564979a8af688dfd453a296619aa97306d1292ac not found: ID does not exist"
Sep 17 16:54:16.883470 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.883434    2565 scope.go:117] "RemoveContainer" containerID="5eb982c83a712915b01b3d02db3c79f1f343a79e27685ca278e962393e0fefea"
Sep 17 16:54:16.883726 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:54:16.883711    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb982c83a712915b01b3d02db3c79f1f343a79e27685ca278e962393e0fefea\": container with ID starting with 5eb982c83a712915b01b3d02db3c79f1f343a79e27685ca278e962393e0fefea not found: ID does not exist" containerID="5eb982c83a712915b01b3d02db3c79f1f343a79e27685ca278e962393e0fefea"
Sep 17 16:54:16.883775 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.883730    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb982c83a712915b01b3d02db3c79f1f343a79e27685ca278e962393e0fefea"} err="failed to get container status \"5eb982c83a712915b01b3d02db3c79f1f343a79e27685ca278e962393e0fefea\": rpc error: code = NotFound desc = could not find container \"5eb982c83a712915b01b3d02db3c79f1f343a79e27685ca278e962393e0fefea\": container with ID starting with 5eb982c83a712915b01b3d02db3c79f1f343a79e27685ca278e962393e0fefea not found: ID does not exist"
Sep 17 16:54:16.883775 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.883745    2565 scope.go:117] "RemoveContainer" containerID="727cd66383f22804614edcff865d978c63842a0307e4e6f6419db117de257021"
Sep 17 16:54:16.884058 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:54:16.884040    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727cd66383f22804614edcff865d978c63842a0307e4e6f6419db117de257021\": container with ID starting with 727cd66383f22804614edcff865d978c63842a0307e4e6f6419db117de257021 not found: ID does not exist" containerID="727cd66383f22804614edcff865d978c63842a0307e4e6f6419db117de257021"
Sep 17 16:54:16.884142 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.884080    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727cd66383f22804614edcff865d978c63842a0307e4e6f6419db117de257021"} err="failed to get container status \"727cd66383f22804614edcff865d978c63842a0307e4e6f6419db117de257021\": rpc error: code = NotFound desc = could not find container \"727cd66383f22804614edcff865d978c63842a0307e4e6f6419db117de257021\": container with ID starting with 727cd66383f22804614edcff865d978c63842a0307e4e6f6419db117de257021 not found: ID does not exist"
Sep 17 16:54:16.884142 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.884101    2565 scope.go:117] "RemoveContainer" containerID="11d31e39e3baa2e2544d7f640cd81dd313c22f78a617ec84b0c4c1306f903b0c"
Sep 17 16:54:16.884357 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:54:16.884338    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d31e39e3baa2e2544d7f640cd81dd313c22f78a617ec84b0c4c1306f903b0c\": container with ID starting with 11d31e39e3baa2e2544d7f640cd81dd313c22f78a617ec84b0c4c1306f903b0c not found: ID does not exist" containerID="11d31e39e3baa2e2544d7f640cd81dd313c22f78a617ec84b0c4c1306f903b0c"
Sep 17 16:54:16.884405 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:16.884361    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d31e39e3baa2e2544d7f640cd81dd313c22f78a617ec84b0c4c1306f903b0c"} err="failed to get container status \"11d31e39e3baa2e2544d7f640cd81dd313c22f78a617ec84b0c4c1306f903b0c\": rpc error: code = NotFound desc = could not find container \"11d31e39e3baa2e2544d7f640cd81dd313c22f78a617ec84b0c4c1306f903b0c\": container with ID starting with 11d31e39e3baa2e2544d7f640cd81dd313c22f78a617ec84b0c4c1306f903b0c not found: ID does not exist"
Sep 17 16:54:18.744519 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:18.744484    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 16:54:18.823768 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:18.823737    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" path="/var/lib/kubelet/pods/50258a5a-f246-47e2-b63c-580c349c0d7f/volumes"
Sep 17 16:54:19.844751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:19.844718    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:54:19.845380 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:19.845350    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 16:54:23.705624 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:23.705586    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 16:54:26.078828 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:26.078799    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:54:26.137374 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:26.137349    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:54:28.744406 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:28.744361    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 16:54:29.846237 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:29.846184    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 16:54:33.706148 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:33.706108    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 16:54:38.744496 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:38.744454    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 16:54:39.845599 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:39.845562    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 16:54:43.706139 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:43.706096    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 16:54:48.744489 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:48.744445    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 16:54:49.845329 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:49.845295    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 16:54:53.705729 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:53.705690    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 16:54:58.744062 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:58.744009    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 16:54:59.845693 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:54:59.845656    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 16:55:03.707117 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:03.707083    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:55:08.745637 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:08.744938    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:55:09.845372 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:09.845334    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 16:55:11.029885 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:11.029852    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"]
Sep 17 16:55:11.030251 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:11.030099    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kserve-container" containerID="cri-o://8377ed2738f73402a2b6c83911c8b108c0391aeacfaf8ba60c661ffa03dfee49" gracePeriod=30
Sep 17 16:55:11.030251 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:11.030135    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kube-rbac-proxy" containerID="cri-o://e909c5aabf722489f149e767a2359ce72bd6ca11c8f91d2404737d0d2bf89c34" gracePeriod=30
Sep 17 16:55:11.052823 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:11.052799    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:12.082861 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:12.082827    2565 generic.go:356] "Generic (PLEG): container finished" podID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerID="e909c5aabf722489f149e767a2359ce72bd6ca11c8f91d2404737d0d2bf89c34" exitCode=2
Sep 17 16:55:12.083225 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:12.082897    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" event={"ID":"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7","Type":"ContainerDied","Data":"e909c5aabf722489f149e767a2359ce72bd6ca11c8f91d2404737d0d2bf89c34"}
Sep 17 16:55:13.700890 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:13.700852    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Sep 17 16:55:13.705564 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:13.705536    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 16:55:14.963801 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:14.963776    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:55:15.015847 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.015786    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kube-rbac-proxy-sar-config\") pod \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") "
Sep 17 16:55:15.015847 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.015839    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-cabundle-cert\") pod \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") "
Sep 17 16:55:15.016053 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.015878    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fghkv\" (UniqueName: \"kubernetes.io/projected/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kube-api-access-fghkv\") pod \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") "
Sep 17 16:55:15.016053 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.015905    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kserve-provision-location\") pod \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") "
Sep 17 16:55:15.016053 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.015957    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-proxy-tls\") pod \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\" (UID: \"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7\") "
Sep 17 16:55:15.016228 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.016200    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" (UID: "ecf6921d-2cbf-4f63-b6b0-3a24c67114d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:55:15.016279 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.016224    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" (UID: "ecf6921d-2cbf-4f63-b6b0-3a24c67114d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:55:15.016279 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.016243    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" (UID: "ecf6921d-2cbf-4f63-b6b0-3a24c67114d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:55:15.017850 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.017825    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" (UID: "ecf6921d-2cbf-4f63-b6b0-3a24c67114d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:55:15.017954 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.017885    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kube-api-access-fghkv" (OuterVolumeSpecName: "kube-api-access-fghkv") pod "ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" (UID: "ecf6921d-2cbf-4f63-b6b0-3a24c67114d7"). InnerVolumeSpecName "kube-api-access-fghkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:55:15.097757 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.097728    2565 generic.go:356] "Generic (PLEG): container finished" podID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerID="8377ed2738f73402a2b6c83911c8b108c0391aeacfaf8ba60c661ffa03dfee49" exitCode=0
Sep 17 16:55:15.097855 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.097790    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"
Sep 17 16:55:15.097855 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.097806    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" event={"ID":"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7","Type":"ContainerDied","Data":"8377ed2738f73402a2b6c83911c8b108c0391aeacfaf8ba60c661ffa03dfee49"}
Sep 17 16:55:15.097945 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.097862    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c" event={"ID":"ecf6921d-2cbf-4f63-b6b0-3a24c67114d7","Type":"ContainerDied","Data":"a067b2bc9a485c6190b2eb96c0f563caef338c8c25efd4696db9873d5ac5d736"}
Sep 17 16:55:15.097945 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.097890    2565 scope.go:117] "RemoveContainer" containerID="e909c5aabf722489f149e767a2359ce72bd6ca11c8f91d2404737d0d2bf89c34"
Sep 17 16:55:15.106810 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.106790    2565 scope.go:117] "RemoveContainer" containerID="8377ed2738f73402a2b6c83911c8b108c0391aeacfaf8ba60c661ffa03dfee49"
Sep 17 16:55:15.113688 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.113669    2565 scope.go:117] "RemoveContainer" containerID="f3cf1b05fc489cdf2387aad168b30dca887343307afe65002656f049a6478d64"
Sep 17 16:55:15.116826 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.116808    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fghkv\" (UniqueName: \"kubernetes.io/projected/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kube-api-access-fghkv\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:55:15.116906 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.116827    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:55:15.116906 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.116837    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:55:15.116906 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.116847    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:55:15.116906 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.116855    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:55:15.120073 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.120053    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"]
Sep 17 16:55:15.121497 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.121481    2565 scope.go:117] "RemoveContainer" containerID="e909c5aabf722489f149e767a2359ce72bd6ca11c8f91d2404737d0d2bf89c34"
Sep 17 16:55:15.121730 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:55:15.121713    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e909c5aabf722489f149e767a2359ce72bd6ca11c8f91d2404737d0d2bf89c34\": container with ID starting with e909c5aabf722489f149e767a2359ce72bd6ca11c8f91d2404737d0d2bf89c34 not found: ID does not exist" containerID="e909c5aabf722489f149e767a2359ce72bd6ca11c8f91d2404737d0d2bf89c34"
Sep 17 16:55:15.121786 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.121737    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e909c5aabf722489f149e767a2359ce72bd6ca11c8f91d2404737d0d2bf89c34"} err="failed to get container status \"e909c5aabf722489f149e767a2359ce72bd6ca11c8f91d2404737d0d2bf89c34\": rpc error: code = NotFound desc = could not find container \"e909c5aabf722489f149e767a2359ce72bd6ca11c8f91d2404737d0d2bf89c34\": container with ID starting with e909c5aabf722489f149e767a2359ce72bd6ca11c8f91d2404737d0d2bf89c34 not found: ID does not exist"
Sep 17 16:55:15.121786 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.121757    2565 scope.go:117] "RemoveContainer" containerID="8377ed2738f73402a2b6c83911c8b108c0391aeacfaf8ba60c661ffa03dfee49"
Sep 17 16:55:15.122055 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:55:15.122035    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8377ed2738f73402a2b6c83911c8b108c0391aeacfaf8ba60c661ffa03dfee49\": container with ID starting with 8377ed2738f73402a2b6c83911c8b108c0391aeacfaf8ba60c661ffa03dfee49 not found: ID does not exist" containerID="8377ed2738f73402a2b6c83911c8b108c0391aeacfaf8ba60c661ffa03dfee49"
Sep 17 16:55:15.122177 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.122063    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8377ed2738f73402a2b6c83911c8b108c0391aeacfaf8ba60c661ffa03dfee49"} err="failed to get container status \"8377ed2738f73402a2b6c83911c8b108c0391aeacfaf8ba60c661ffa03dfee49\": rpc error: code = NotFound desc = could not find container \"8377ed2738f73402a2b6c83911c8b108c0391aeacfaf8ba60c661ffa03dfee49\": container with ID starting with 8377ed2738f73402a2b6c83911c8b108c0391aeacfaf8ba60c661ffa03dfee49 not found: ID does not exist"
Sep 17 16:55:15.122177 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.122085    2565 scope.go:117] "RemoveContainer" containerID="f3cf1b05fc489cdf2387aad168b30dca887343307afe65002656f049a6478d64"
Sep 17 16:55:15.122753 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:55:15.122726    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3cf1b05fc489cdf2387aad168b30dca887343307afe65002656f049a6478d64\": container with ID starting with f3cf1b05fc489cdf2387aad168b30dca887343307afe65002656f049a6478d64 not found: ID does not exist" containerID="f3cf1b05fc489cdf2387aad168b30dca887343307afe65002656f049a6478d64"
Sep 17 16:55:15.122843 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.122758    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3cf1b05fc489cdf2387aad168b30dca887343307afe65002656f049a6478d64"} err="failed to get container status \"f3cf1b05fc489cdf2387aad168b30dca887343307afe65002656f049a6478d64\": rpc error: code = NotFound desc = could not find container \"f3cf1b05fc489cdf2387aad168b30dca887343307afe65002656f049a6478d64\": container with ID starting with f3cf1b05fc489cdf2387aad168b30dca887343307afe65002656f049a6478d64 not found: ID does not exist"
Sep 17 16:55:15.124176 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:15.124154    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-696c5cc6cc-rwk9c"]
Sep 17 16:55:16.183534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.183500    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:16.214978 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.214953    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"]
Sep 17 16:55:16.215270 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215259    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="storage-initializer"
Sep 17 16:55:16.215312 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215272    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="storage-initializer"
Sep 17 16:55:16.215312 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215280    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="storage-initializer"
Sep 17 16:55:16.215312 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215286    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="storage-initializer"
Sep 17 16:55:16.215312 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215299    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container"
Sep 17 16:55:16.215312 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215305    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container"
Sep 17 16:55:16.215465 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215317    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent"
Sep 17 16:55:16.215465 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215322    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent"
Sep 17 16:55:16.215465 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215334    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kube-rbac-proxy"
Sep 17 16:55:16.215465 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215340    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kube-rbac-proxy"
Sep 17 16:55:16.215465 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215347    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kserve-container"
Sep 17 16:55:16.215465 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215352    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kserve-container"
Sep 17 16:55:16.215465 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215357    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kube-rbac-proxy"
Sep 17 16:55:16.215465 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215362    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kube-rbac-proxy"
Sep 17 16:55:16.215465 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215410    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kube-rbac-proxy"
Sep 17 16:55:16.215465 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215417    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="kserve-container"
Sep 17 16:55:16.215465 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215424    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kserve-container"
Sep 17 16:55:16.215465 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215430    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="50258a5a-f246-47e2-b63c-580c349c0d7f" containerName="agent"
Sep 17 16:55:16.215465 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.215438    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" containerName="kube-rbac-proxy"
Sep 17 16:55:16.218589 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.218573    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.220977 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.220958    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 16:55:16.221100 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.221085    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 16:55:16.227469 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.227443    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"]
Sep 17 16:55:16.256874 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.256853    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:16.304685 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.304661    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"]
Sep 17 16:55:16.306997 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.306983    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.309831 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.309815    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 16:55:16.325136 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.325117    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72bfac89-404c-41e1-9f64-88bee468104f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-np76t\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.325225 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.325145    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72bfac89-404c-41e1-9f64-88bee468104f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-np76t\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.325225 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.325169    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72bfac89-404c-41e1-9f64-88bee468104f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-np76t\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.325225 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.325187    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grtpn\" (UniqueName: \"kubernetes.io/projected/72bfac89-404c-41e1-9f64-88bee468104f-kube-api-access-grtpn\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-np76t\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.325330 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.325253    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72bfac89-404c-41e1-9f64-88bee468104f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-np76t\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.332163 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.332145    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"]
Sep 17 16:55:16.398835 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.398797    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:16.426495 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.426474    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2a31cd0-50fd-4891-a27d-e85645918526-proxy-tls\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-bwhcb\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.426595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.426511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72bfac89-404c-41e1-9f64-88bee468104f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-np76t\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.426595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.426527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2a31cd0-50fd-4891-a27d-e85645918526-cabundle-cert\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-bwhcb\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.426595 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.426544    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72bfac89-404c-41e1-9f64-88bee468104f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-np76t\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.426762 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.426591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72bfac89-404c-41e1-9f64-88bee468104f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-np76t\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.426762 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.426623    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49f9m\" (UniqueName: \"kubernetes.io/projected/b2a31cd0-50fd-4891-a27d-e85645918526-kube-api-access-49f9m\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-bwhcb\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.426762 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.426655    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grtpn\" (UniqueName: \"kubernetes.io/projected/72bfac89-404c-41e1-9f64-88bee468104f-kube-api-access-grtpn\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-np76t\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.426762 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.426740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72bfac89-404c-41e1-9f64-88bee468104f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-np76t\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.426973 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.426783    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2a31cd0-50fd-4891-a27d-e85645918526-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-bwhcb\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.426973 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.426812    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2a31cd0-50fd-4891-a27d-e85645918526-kserve-provision-location\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-bwhcb\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.427158 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.427137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72bfac89-404c-41e1-9f64-88bee468104f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-np76t\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.427348 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.427329    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72bfac89-404c-41e1-9f64-88bee468104f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-np76t\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.427388 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.427329    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72bfac89-404c-41e1-9f64-88bee468104f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-np76t\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.428893 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.428876    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72bfac89-404c-41e1-9f64-88bee468104f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-np76t\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.435214 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.435171    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grtpn\" (UniqueName: \"kubernetes.io/projected/72bfac89-404c-41e1-9f64-88bee468104f-kube-api-access-grtpn\") pod \"isvc-canary-stop-v2-predictor-775495c6b4-np76t\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.487348 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.487326    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:16.528057 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.528028    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49f9m\" (UniqueName: \"kubernetes.io/projected/b2a31cd0-50fd-4891-a27d-e85645918526-kube-api-access-49f9m\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-bwhcb\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.528159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.528072    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2a31cd0-50fd-4891-a27d-e85645918526-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-bwhcb\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.528159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.528089    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2a31cd0-50fd-4891-a27d-e85645918526-kserve-provision-location\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-bwhcb\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.528159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.528126    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2a31cd0-50fd-4891-a27d-e85645918526-proxy-tls\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-bwhcb\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.528159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.528152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2a31cd0-50fd-4891-a27d-e85645918526-cabundle-cert\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-bwhcb\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.528393 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.528372    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:16.528575 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.528555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2a31cd0-50fd-4891-a27d-e85645918526-kserve-provision-location\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-bwhcb\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.528711 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.528694    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2a31cd0-50fd-4891-a27d-e85645918526-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-bwhcb\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.528776 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.528760    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2a31cd0-50fd-4891-a27d-e85645918526-cabundle-cert\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-bwhcb\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.530489 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.530467    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2a31cd0-50fd-4891-a27d-e85645918526-proxy-tls\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-bwhcb\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.535393 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.535370    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49f9m\" (UniqueName: \"kubernetes.io/projected/b2a31cd0-50fd-4891-a27d-e85645918526-kube-api-access-49f9m\") pod \"isvc-canary-stop-predictor-67d6bcd4b5-bwhcb\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.616364 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.616325    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:16.798655 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.798627    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"]
Sep 17 16:55:16.799950 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:55:16.799926    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bfac89_404c_41e1_9f64_88bee468104f.slice/crio-8b7b4eb82842cdeda9694fe109a4911872e968dedf233b9b88c56bf8e645a635 WatchSource:0}: Error finding container 8b7b4eb82842cdeda9694fe109a4911872e968dedf233b9b88c56bf8e645a635: Status 404 returned error can't find the container with id 8b7b4eb82842cdeda9694fe109a4911872e968dedf233b9b88c56bf8e645a635
Sep 17 16:55:16.822761 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.822737    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecf6921d-2cbf-4f63-b6b0-3a24c67114d7" path="/var/lib/kubelet/pods/ecf6921d-2cbf-4f63-b6b0-3a24c67114d7/volumes"
Sep 17 16:55:16.885258 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:16.885238    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"]
Sep 17 16:55:16.886925 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:55:16.886895    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a31cd0_50fd_4891_a27d_e85645918526.slice/crio-6b86aed703d96a2f7f0af8fbb76f936f66d2d7ed259e7d39b12c5c280a0e3e13 WatchSource:0}: Error finding container 6b86aed703d96a2f7f0af8fbb76f936f66d2d7ed259e7d39b12c5c280a0e3e13: Status 404 returned error can't find the container with id 6b86aed703d96a2f7f0af8fbb76f936f66d2d7ed259e7d39b12c5c280a0e3e13
Sep 17 16:55:17.108354 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:17.108252    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" event={"ID":"72bfac89-404c-41e1-9f64-88bee468104f","Type":"ContainerStarted","Data":"b5f52050130d90f09b84f0848cdbc3ae968a561924d5d9b5de62421925543617"}
Sep 17 16:55:17.108354 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:17.108329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" event={"ID":"72bfac89-404c-41e1-9f64-88bee468104f","Type":"ContainerStarted","Data":"8b7b4eb82842cdeda9694fe109a4911872e968dedf233b9b88c56bf8e645a635"}
Sep 17 16:55:17.111431 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:17.111402    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" event={"ID":"b2a31cd0-50fd-4891-a27d-e85645918526","Type":"ContainerStarted","Data":"19b36a6a8a60020e40f51c3d272eb7e56c9df14952537e1f222f43c070ca2cca"}
Sep 17 16:55:17.111537 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:17.111464    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" event={"ID":"b2a31cd0-50fd-4891-a27d-e85645918526","Type":"ContainerStarted","Data":"6b86aed703d96a2f7f0af8fbb76f936f66d2d7ed259e7d39b12c5c280a0e3e13"}
Sep 17 16:55:19.846091 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:19.846057    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:55:21.132903 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:21.132870    2565 generic.go:356] "Generic (PLEG): container finished" podID="72bfac89-404c-41e1-9f64-88bee468104f" containerID="b5f52050130d90f09b84f0848cdbc3ae968a561924d5d9b5de62421925543617" exitCode=0
Sep 17 16:55:21.133265 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:21.132951    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" event={"ID":"72bfac89-404c-41e1-9f64-88bee468104f","Type":"ContainerDied","Data":"b5f52050130d90f09b84f0848cdbc3ae968a561924d5d9b5de62421925543617"}
Sep 17 16:55:21.134585 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:21.134560    2565 generic.go:356] "Generic (PLEG): container finished" podID="b2a31cd0-50fd-4891-a27d-e85645918526" containerID="19b36a6a8a60020e40f51c3d272eb7e56c9df14952537e1f222f43c070ca2cca" exitCode=0
Sep 17 16:55:21.134774 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:21.134628    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" event={"ID":"b2a31cd0-50fd-4891-a27d-e85645918526","Type":"ContainerDied","Data":"19b36a6a8a60020e40f51c3d272eb7e56c9df14952537e1f222f43c070ca2cca"}
Sep 17 16:55:22.141762 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:22.141726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" event={"ID":"72bfac89-404c-41e1-9f64-88bee468104f","Type":"ContainerStarted","Data":"465a0538ae60044a453521286e7b892572b0252e785ba41531ee16eae34cf465"}
Sep 17 16:55:22.142161 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:22.141790    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" event={"ID":"72bfac89-404c-41e1-9f64-88bee468104f","Type":"ContainerStarted","Data":"3ec51af7d8e3edda7d3c1cd552d8f0d1926e447119bdf3d2fe4a757c61886653"}
Sep 17 16:55:22.142161 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:22.142150    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:22.142352 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:22.142332    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:22.143791 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:22.143760    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 16:55:22.144071 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:22.144048    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" event={"ID":"b2a31cd0-50fd-4891-a27d-e85645918526","Type":"ContainerStarted","Data":"d9e42961a625200fd448e3ac8bf62650d784978c1fec6a466a7967fb422bd818"}
Sep 17 16:55:22.144128 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:22.144102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" event={"ID":"b2a31cd0-50fd-4891-a27d-e85645918526","Type":"ContainerStarted","Data":"ce662c7f42b697487532ecb893e5ce09cb5a9738e365c13792a0cad4dafce647"}
Sep 17 16:55:22.144386 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:22.144372    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:22.144423 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:22.144413    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:22.145569 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:22.145546    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 16:55:22.160378 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:22.160333    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" podStartSLOduration=2.174223954 podStartE2EDuration="6.16032009s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.986096136s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:55:22.159248555 +0000 UTC m=+787.725360331" watchObservedRunningTime="2026-09-17 16:55:22.16032009 +0000 UTC m=+787.726431802"
Sep 17 16:55:22.177173 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:22.177138    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" podStartSLOduration=2.250700234 podStartE2EDuration="6.177125015s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.926424781s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:55:22.175689684 +0000 UTC m=+787.741801398" watchObservedRunningTime="2026-09-17 16:55:22.177125015 +0000 UTC m=+787.743236737"
Sep 17 16:55:23.149252 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:23.149209    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 16:55:23.149699 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:23.149369    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 16:55:26.118517 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:26.118489    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:26.274850 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:26.274810    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:28.154116 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:28.154084    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:55:28.154576 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:28.154517    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:55:28.154996 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:28.154966    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 16:55:28.155126 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:28.155000    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 16:55:36.211094 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.211063    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:36.282130 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.282096    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"]
Sep 17 16:55:36.282352 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.282324    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kserve-container" containerID="cri-o://f64f8c854f58a2ebbb35c612b600566e38e35a6277de5aa81198c12610352ccf" gracePeriod=30
Sep 17 16:55:36.282496 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.282362    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kube-rbac-proxy" containerID="cri-o://b38bab56edf049a3d6212be966d0895c29355a78c5f45044276708ef4cf66a75" gracePeriod=30
Sep 17 16:55:36.310184 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.310158    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:36.319486 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.319462    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"]
Sep 17 16:55:36.319797 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.319751    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kserve-container" containerID="cri-o://ec4f1cefd8417a22739855d925637e75dffda70f1525dc67951929dd24cd8ed9" gracePeriod=30
Sep 17 16:55:36.319869 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.319824    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kube-rbac-proxy" containerID="cri-o://a7af0a521c45cdd317ed3c040c5402b31f4dbf07dbd4eb99997a1f17984ee7f0" gracePeriod=30
Sep 17 16:55:36.411747 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.411716    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:36.420988 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.420961    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"]
Sep 17 16:55:36.424570 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.424548    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:36.427103 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.427079    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-48552-predictor-serving-cert\""
Sep 17 16:55:36.427197 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.427129    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-48552-kube-rbac-proxy-sar-config\""
Sep 17 16:55:36.434736 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.434715    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"]
Sep 17 16:55:36.479815 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.479743    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkgd2\" (UniqueName: \"kubernetes.io/projected/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kube-api-access-vkgd2\") pod \"isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:36.479815 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.479802    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:36.480016 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.479834    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:36.480016 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.479852    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:36.480016 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.479925    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:36.509751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.509722    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:36.520301 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.520277    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"]
Sep 17 16:55:36.523594 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.523578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:36.525861 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.525842    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-48552-predictor-serving-cert\""
Sep 17 16:55:36.525978 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.525883    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-48552-kube-rbac-proxy-sar-config\""
Sep 17 16:55:36.536448 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.536426    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"]
Sep 17 16:55:36.569252 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.569222    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:36.580409 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.580389    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsff4\" (UniqueName: \"kubernetes.io/projected/47703c88-87a0-4e64-94f9-2306d99465c5-kube-api-access-gsff4\") pod \"isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:36.580486 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.580426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47703c88-87a0-4e64-94f9-2306d99465c5-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:36.580486 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.580460    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:36.580558 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.580513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47703c88-87a0-4e64-94f9-2306d99465c5-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:36.580558 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:55:36.580540    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-48552-predictor-serving-cert" not found
Sep 17 16:55:36.580558 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.580548    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:36.580665 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.580566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47703c88-87a0-4e64-94f9-2306d99465c5-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:36.580665 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:55:36.580587    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-proxy-tls podName:cb36dbe6-9d3f-418e-bfa9-43ec3862f119 nodeName:}" failed. No retries permitted until 2026-09-17 16:55:37.080573362 +0000 UTC m=+802.646685062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" (UID: "cb36dbe6-9d3f-418e-bfa9-43ec3862f119") : secret "isvc-sklearn-graph-raw-hpa-48552-predictor-serving-cert" not found
Sep 17 16:55:36.580665 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.580628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:36.580799 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.580665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:36.580799 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.580686    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47703c88-87a0-4e64-94f9-2306d99465c5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:36.580799 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.580742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkgd2\" (UniqueName: \"kubernetes.io/projected/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kube-api-access-vkgd2\") pod \"isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:36.580899 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.580883    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:36.581323 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.581307    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:36.581419 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.581403    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:36.589641 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.589614    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkgd2\" (UniqueName: \"kubernetes.io/projected/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kube-api-access-vkgd2\") pod \"isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:36.681160 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.681131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsff4\" (UniqueName: \"kubernetes.io/projected/47703c88-87a0-4e64-94f9-2306d99465c5-kube-api-access-gsff4\") pod \"isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:36.681160 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.681165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47703c88-87a0-4e64-94f9-2306d99465c5-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:36.681381 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.681225    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47703c88-87a0-4e64-94f9-2306d99465c5-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:36.681381 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.681252    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47703c88-87a0-4e64-94f9-2306d99465c5-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:36.681381 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:55:36.681365    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-48552-predictor-serving-cert" not found
Sep 17 16:55:36.681537 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.681404    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47703c88-87a0-4e64-94f9-2306d99465c5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:36.681537 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:55:36.681468    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47703c88-87a0-4e64-94f9-2306d99465c5-proxy-tls podName:47703c88-87a0-4e64-94f9-2306d99465c5 nodeName:}" failed. No retries permitted until 2026-09-17 16:55:37.18144716 +0000 UTC m=+802.747558860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/47703c88-87a0-4e64-94f9-2306d99465c5-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" (UID: "47703c88-87a0-4e64-94f9-2306d99465c5") : secret "isvc-xgboost-graph-raw-hpa-48552-predictor-serving-cert" not found
Sep 17 16:55:36.681644 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.681578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47703c88-87a0-4e64-94f9-2306d99465c5-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:36.682046 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.682026    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47703c88-87a0-4e64-94f9-2306d99465c5-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:36.682120 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.682046    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47703c88-87a0-4e64-94f9-2306d99465c5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:36.689301 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.689277    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsff4\" (UniqueName: \"kubernetes.io/projected/47703c88-87a0-4e64-94f9-2306d99465c5-kube-api-access-gsff4\") pod \"isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:36.810428 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.810360    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:36.931283 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:36.931256    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:55:37.085330 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:37.085259    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:37.087681 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:37.087661    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:37.185989 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:37.185961    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47703c88-87a0-4e64-94f9-2306d99465c5-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:37.188205 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:37.188186    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47703c88-87a0-4e64-94f9-2306d99465c5-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:37.215588 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:37.215564    2565 generic.go:356] "Generic (PLEG): container finished" podID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerID="a7af0a521c45cdd317ed3c040c5402b31f4dbf07dbd4eb99997a1f17984ee7f0" exitCode=2
Sep 17 16:55:37.215897 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:37.215632    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" event={"ID":"8f52e550-2266-464e-bbc1-b12e1b7f3f55","Type":"ContainerDied","Data":"a7af0a521c45cdd317ed3c040c5402b31f4dbf07dbd4eb99997a1f17984ee7f0"}
Sep 17 16:55:37.217685 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:37.217666    2565 generic.go:356] "Generic (PLEG): container finished" podID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerID="b38bab56edf049a3d6212be966d0895c29355a78c5f45044276708ef4cf66a75" exitCode=2
Sep 17 16:55:37.217771 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:37.217692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" event={"ID":"71333a7b-dba5-4488-8dfc-930854c8dfe3","Type":"ContainerDied","Data":"b38bab56edf049a3d6212be966d0895c29355a78c5f45044276708ef4cf66a75"}
Sep 17 16:55:37.336891 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:37.336836    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:37.433558 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:37.433529    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:37.608839 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:37.608795    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"]
Sep 17 16:55:37.610932 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:55:37.610887    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb36dbe6_9d3f_418e_bfa9_43ec3862f119.slice/crio-ef4723be7cab4b867466ea7ecc905b7ae5b26606151801f4acb9aa388f0c9ede WatchSource:0}: Error finding container ef4723be7cab4b867466ea7ecc905b7ae5b26606151801f4acb9aa388f0c9ede: Status 404 returned error can't find the container with id ef4723be7cab4b867466ea7ecc905b7ae5b26606151801f4acb9aa388f0c9ede
Sep 17 16:55:37.703257 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:37.703229    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"]
Sep 17 16:55:37.706302 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:55:37.706277    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47703c88_87a0_4e64_94f9_2306d99465c5.slice/crio-3953c5fdfb719a789384de28f8c0a7d5ebbe30e62937cd7c9e0169991046ebc3 WatchSource:0}: Error finding container 3953c5fdfb719a789384de28f8c0a7d5ebbe30e62937cd7c9e0169991046ebc3: Status 404 returned error can't find the container with id 3953c5fdfb719a789384de28f8c0a7d5ebbe30e62937cd7c9e0169991046ebc3
Sep 17 16:55:38.155354 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:38.155229    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 16:55:38.155354 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:38.155280    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 16:55:38.224871 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:38.224841    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" event={"ID":"cb36dbe6-9d3f-418e-bfa9-43ec3862f119","Type":"ContainerStarted","Data":"dac0382c5aec9bbe8fcfcfb0111b08096b614e938cca1400e7326a8b83e43916"}
Sep 17 16:55:38.225325 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:38.224901    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" event={"ID":"cb36dbe6-9d3f-418e-bfa9-43ec3862f119","Type":"ContainerStarted","Data":"ef4723be7cab4b867466ea7ecc905b7ae5b26606151801f4acb9aa388f0c9ede"}
Sep 17 16:55:38.226682 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:38.226653    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" event={"ID":"47703c88-87a0-4e64-94f9-2306d99465c5","Type":"ContainerStarted","Data":"ddea334e743dc15d796a1c9d4f9c257bd4fff2699fdb0c3db6c1db27615308b2"}
Sep 17 16:55:38.226782 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:38.226713    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" event={"ID":"47703c88-87a0-4e64-94f9-2306d99465c5","Type":"ContainerStarted","Data":"3953c5fdfb719a789384de28f8c0a7d5ebbe30e62937cd7c9e0169991046ebc3"}
Sep 17 16:55:38.738652 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:38.738612    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 17 16:55:38.743814 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:38.743782    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 16:55:39.841622 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:39.841581    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 17 16:55:39.846108 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:39.846069    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 16:55:40.238310 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.238281    2565 generic.go:356] "Generic (PLEG): container finished" podID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerID="ec4f1cefd8417a22739855d925637e75dffda70f1525dc67951929dd24cd8ed9" exitCode=0
Sep 17 16:55:40.238416 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.238351    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" event={"ID":"8f52e550-2266-464e-bbc1-b12e1b7f3f55","Type":"ContainerDied","Data":"ec4f1cefd8417a22739855d925637e75dffda70f1525dc67951929dd24cd8ed9"}
Sep 17 16:55:40.369972 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.369949    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:55:40.515546 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.515469    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kube-rbac-proxy-sar-config\") pod \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") "
Sep 17 16:55:40.515546 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.515514    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kserve-provision-location\") pod \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") "
Sep 17 16:55:40.515546 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.515533    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phvph\" (UniqueName: \"kubernetes.io/projected/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kube-api-access-phvph\") pod \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") "
Sep 17 16:55:40.515819 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.515553    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f52e550-2266-464e-bbc1-b12e1b7f3f55-cabundle-cert\") pod \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") "
Sep 17 16:55:40.515819 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.515617    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f52e550-2266-464e-bbc1-b12e1b7f3f55-proxy-tls\") pod \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\" (UID: \"8f52e550-2266-464e-bbc1-b12e1b7f3f55\") "
Sep 17 16:55:40.515951 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.515852    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f52e550-2266-464e-bbc1-b12e1b7f3f55" (UID: "8f52e550-2266-464e-bbc1-b12e1b7f3f55"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:55:40.515951 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.515877    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f52e550-2266-464e-bbc1-b12e1b7f3f55" (UID: "8f52e550-2266-464e-bbc1-b12e1b7f3f55"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:55:40.516059 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.515977    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f52e550-2266-464e-bbc1-b12e1b7f3f55-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f52e550-2266-464e-bbc1-b12e1b7f3f55" (UID: "8f52e550-2266-464e-bbc1-b12e1b7f3f55"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:55:40.517573 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.517553    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kube-api-access-phvph" (OuterVolumeSpecName: "kube-api-access-phvph") pod "8f52e550-2266-464e-bbc1-b12e1b7f3f55" (UID: "8f52e550-2266-464e-bbc1-b12e1b7f3f55"). InnerVolumeSpecName "kube-api-access-phvph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:55:40.517732 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.517716    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f52e550-2266-464e-bbc1-b12e1b7f3f55-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f52e550-2266-464e-bbc1-b12e1b7f3f55" (UID: "8f52e550-2266-464e-bbc1-b12e1b7f3f55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:55:40.616863 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.616828    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f52e550-2266-464e-bbc1-b12e1b7f3f55-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:55:40.616863 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.616857    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:55:40.616863 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.616866    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:55:40.617096 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.616874    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phvph\" (UniqueName: \"kubernetes.io/projected/8f52e550-2266-464e-bbc1-b12e1b7f3f55-kube-api-access-phvph\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:55:40.617096 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.616883    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f52e550-2266-464e-bbc1-b12e1b7f3f55-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:55:40.813115 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.813093    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:55:40.918534 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.918505    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71333a7b-dba5-4488-8dfc-930854c8dfe3-kserve-provision-location\") pod \"71333a7b-dba5-4488-8dfc-930854c8dfe3\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") "
Sep 17 16:55:40.918977 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.918571    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71333a7b-dba5-4488-8dfc-930854c8dfe3-proxy-tls\") pod \"71333a7b-dba5-4488-8dfc-930854c8dfe3\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") "
Sep 17 16:55:40.918977 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.918598    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71333a7b-dba5-4488-8dfc-930854c8dfe3-kube-rbac-proxy-sar-config\") pod \"71333a7b-dba5-4488-8dfc-930854c8dfe3\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") "
Sep 17 16:55:40.918977 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.918615    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71333a7b-dba5-4488-8dfc-930854c8dfe3-cabundle-cert\") pod \"71333a7b-dba5-4488-8dfc-930854c8dfe3\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") "
Sep 17 16:55:40.918977 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.918655    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvbb7\" (UniqueName: \"kubernetes.io/projected/71333a7b-dba5-4488-8dfc-930854c8dfe3-kube-api-access-vvbb7\") pod \"71333a7b-dba5-4488-8dfc-930854c8dfe3\" (UID: \"71333a7b-dba5-4488-8dfc-930854c8dfe3\") "
Sep 17 16:55:40.918977 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.918867    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71333a7b-dba5-4488-8dfc-930854c8dfe3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71333a7b-dba5-4488-8dfc-930854c8dfe3" (UID: "71333a7b-dba5-4488-8dfc-930854c8dfe3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:55:40.919171 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.919015    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71333a7b-dba5-4488-8dfc-930854c8dfe3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "71333a7b-dba5-4488-8dfc-930854c8dfe3" (UID: "71333a7b-dba5-4488-8dfc-930854c8dfe3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:55:40.919171 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.919011    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71333a7b-dba5-4488-8dfc-930854c8dfe3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71333a7b-dba5-4488-8dfc-930854c8dfe3" (UID: "71333a7b-dba5-4488-8dfc-930854c8dfe3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:55:40.920650 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.920629    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71333a7b-dba5-4488-8dfc-930854c8dfe3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71333a7b-dba5-4488-8dfc-930854c8dfe3" (UID: "71333a7b-dba5-4488-8dfc-930854c8dfe3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:55:40.920710 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:40.920659    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71333a7b-dba5-4488-8dfc-930854c8dfe3-kube-api-access-vvbb7" (OuterVolumeSpecName: "kube-api-access-vvbb7") pod "71333a7b-dba5-4488-8dfc-930854c8dfe3" (UID: "71333a7b-dba5-4488-8dfc-930854c8dfe3"). InnerVolumeSpecName "kube-api-access-vvbb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:55:41.020113 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.020062    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71333a7b-dba5-4488-8dfc-930854c8dfe3-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:55:41.020113 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.020105    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71333a7b-dba5-4488-8dfc-930854c8dfe3-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:55:41.020113 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.020116    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71333a7b-dba5-4488-8dfc-930854c8dfe3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:55:41.020113 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.020124    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71333a7b-dba5-4488-8dfc-930854c8dfe3-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:55:41.020397 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.020135    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvbb7\" (UniqueName: \"kubernetes.io/projected/71333a7b-dba5-4488-8dfc-930854c8dfe3-kube-api-access-vvbb7\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:55:41.245606 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.245569    2565 generic.go:356] "Generic (PLEG): container finished" podID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerID="f64f8c854f58a2ebbb35c612b600566e38e35a6277de5aa81198c12610352ccf" exitCode=0
Sep 17 16:55:41.245801 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.245633    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"
Sep 17 16:55:41.245801 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.245646    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" event={"ID":"71333a7b-dba5-4488-8dfc-930854c8dfe3","Type":"ContainerDied","Data":"f64f8c854f58a2ebbb35c612b600566e38e35a6277de5aa81198c12610352ccf"}
Sep 17 16:55:41.245801 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.245709    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r" event={"ID":"71333a7b-dba5-4488-8dfc-930854c8dfe3","Type":"ContainerDied","Data":"a06a029fefb3f6db06eb575e105f1eedeb5e653a0a6fdba21f90c43d48d42ef0"}
Sep 17 16:55:41.245801 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.245736    2565 scope.go:117] "RemoveContainer" containerID="b38bab56edf049a3d6212be966d0895c29355a78c5f45044276708ef4cf66a75"
Sep 17 16:55:41.248094 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.248069    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk" event={"ID":"8f52e550-2266-464e-bbc1-b12e1b7f3f55","Type":"ContainerDied","Data":"fdd7b904f0c2a1a810ac3993d00d11b9bf708b00821892e35e07974998933350"}
Sep 17 16:55:41.248094 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.248084    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"
Sep 17 16:55:41.256684 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.256668    2565 scope.go:117] "RemoveContainer" containerID="f64f8c854f58a2ebbb35c612b600566e38e35a6277de5aa81198c12610352ccf"
Sep 17 16:55:41.263493 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.263471    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"]
Sep 17 16:55:41.268089 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.268068    2565 scope.go:117] "RemoveContainer" containerID="757684746c8e2f60dcb4b7f4c8a3086b5741f6427388ea52e5d42bcff117cfab"
Sep 17 16:55:41.268213 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.268194    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-48e77-predictor-5477d668d5-pjxzk"]
Sep 17 16:55:41.275291 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.275274    2565 scope.go:117] "RemoveContainer" containerID="b38bab56edf049a3d6212be966d0895c29355a78c5f45044276708ef4cf66a75"
Sep 17 16:55:41.275560 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:55:41.275544    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38bab56edf049a3d6212be966d0895c29355a78c5f45044276708ef4cf66a75\": container with ID starting with b38bab56edf049a3d6212be966d0895c29355a78c5f45044276708ef4cf66a75 not found: ID does not exist" containerID="b38bab56edf049a3d6212be966d0895c29355a78c5f45044276708ef4cf66a75"
Sep 17 16:55:41.275618 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.275570    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38bab56edf049a3d6212be966d0895c29355a78c5f45044276708ef4cf66a75"} err="failed to get container status \"b38bab56edf049a3d6212be966d0895c29355a78c5f45044276708ef4cf66a75\": rpc error: code = NotFound desc = could not find container \"b38bab56edf049a3d6212be966d0895c29355a78c5f45044276708ef4cf66a75\": container with ID starting with b38bab56edf049a3d6212be966d0895c29355a78c5f45044276708ef4cf66a75 not found: ID does not exist"
Sep 17 16:55:41.275618 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.275587    2565 scope.go:117] "RemoveContainer" containerID="f64f8c854f58a2ebbb35c612b600566e38e35a6277de5aa81198c12610352ccf"
Sep 17 16:55:41.275846 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:55:41.275828    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f64f8c854f58a2ebbb35c612b600566e38e35a6277de5aa81198c12610352ccf\": container with ID starting with f64f8c854f58a2ebbb35c612b600566e38e35a6277de5aa81198c12610352ccf not found: ID does not exist" containerID="f64f8c854f58a2ebbb35c612b600566e38e35a6277de5aa81198c12610352ccf"
Sep 17 16:55:41.275898 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.275851    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64f8c854f58a2ebbb35c612b600566e38e35a6277de5aa81198c12610352ccf"} err="failed to get container status \"f64f8c854f58a2ebbb35c612b600566e38e35a6277de5aa81198c12610352ccf\": rpc error: code = NotFound desc = could not find container \"f64f8c854f58a2ebbb35c612b600566e38e35a6277de5aa81198c12610352ccf\": container with ID starting with f64f8c854f58a2ebbb35c612b600566e38e35a6277de5aa81198c12610352ccf not found: ID does not exist"
Sep 17 16:55:41.275898 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.275865    2565 scope.go:117] "RemoveContainer" containerID="757684746c8e2f60dcb4b7f4c8a3086b5741f6427388ea52e5d42bcff117cfab"
Sep 17 16:55:41.276125 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:55:41.276106    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"757684746c8e2f60dcb4b7f4c8a3086b5741f6427388ea52e5d42bcff117cfab\": container with ID starting with 757684746c8e2f60dcb4b7f4c8a3086b5741f6427388ea52e5d42bcff117cfab not found: ID does not exist" containerID="757684746c8e2f60dcb4b7f4c8a3086b5741f6427388ea52e5d42bcff117cfab"
Sep 17 16:55:41.276165 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.276131    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"757684746c8e2f60dcb4b7f4c8a3086b5741f6427388ea52e5d42bcff117cfab"} err="failed to get container status \"757684746c8e2f60dcb4b7f4c8a3086b5741f6427388ea52e5d42bcff117cfab\": rpc error: code = NotFound desc = could not find container \"757684746c8e2f60dcb4b7f4c8a3086b5741f6427388ea52e5d42bcff117cfab\": container with ID starting with 757684746c8e2f60dcb4b7f4c8a3086b5741f6427388ea52e5d42bcff117cfab not found: ID does not exist"
Sep 17 16:55:41.276165 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.276145    2565 scope.go:117] "RemoveContainer" containerID="a7af0a521c45cdd317ed3c040c5402b31f4dbf07dbd4eb99997a1f17984ee7f0"
Sep 17 16:55:41.279551 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.279531    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"]
Sep 17 16:55:41.282097 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.282076    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-48e77-predictor-759d457548-lpz9r"]
Sep 17 16:55:41.283758 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.283742    2565 scope.go:117] "RemoveContainer" containerID="ec4f1cefd8417a22739855d925637e75dffda70f1525dc67951929dd24cd8ed9"
Sep 17 16:55:41.290759 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:41.290742    2565 scope.go:117] "RemoveContainer" containerID="9122d24944c370c959a80944f40c7f1bc6c9638166c2a9939f698f41ed96c0b9"
Sep 17 16:55:42.253279 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:42.253249    2565 generic.go:356] "Generic (PLEG): container finished" podID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerID="dac0382c5aec9bbe8fcfcfb0111b08096b614e938cca1400e7326a8b83e43916" exitCode=0
Sep 17 16:55:42.253715 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:42.253330    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" event={"ID":"cb36dbe6-9d3f-418e-bfa9-43ec3862f119","Type":"ContainerDied","Data":"dac0382c5aec9bbe8fcfcfb0111b08096b614e938cca1400e7326a8b83e43916"}
Sep 17 16:55:42.255042 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:42.255020    2565 generic.go:356] "Generic (PLEG): container finished" podID="47703c88-87a0-4e64-94f9-2306d99465c5" containerID="ddea334e743dc15d796a1c9d4f9c257bd4fff2699fdb0c3db6c1db27615308b2" exitCode=0
Sep 17 16:55:42.255169 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:42.255092    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" event={"ID":"47703c88-87a0-4e64-94f9-2306d99465c5","Type":"ContainerDied","Data":"ddea334e743dc15d796a1c9d4f9c257bd4fff2699fdb0c3db6c1db27615308b2"}
Sep 17 16:55:42.823979 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:42.823940    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" path="/var/lib/kubelet/pods/71333a7b-dba5-4488-8dfc-930854c8dfe3/volumes"
Sep 17 16:55:42.824547 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:42.824525    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" path="/var/lib/kubelet/pods/8f52e550-2266-464e-bbc1-b12e1b7f3f55/volumes"
Sep 17 16:55:43.264444 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:43.264412    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" event={"ID":"cb36dbe6-9d3f-418e-bfa9-43ec3862f119","Type":"ContainerStarted","Data":"7f5378300a7f42aa930c6b17fe54f7c83d0e08d2fb38f99e282a9bf3c21d9990"}
Sep 17 16:55:43.264865 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:43.264472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" event={"ID":"cb36dbe6-9d3f-418e-bfa9-43ec3862f119","Type":"ContainerStarted","Data":"b3100e5ade74b07bd241956658b23c2ba4caa6c1c1c250744d0f2ad1e4939e22"}
Sep 17 16:55:43.264865 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:43.264696    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:43.266559 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:43.266538    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" event={"ID":"47703c88-87a0-4e64-94f9-2306d99465c5","Type":"ContainerStarted","Data":"92350be3353780e857ca5d74c608814dca0bde6e964afba6bed05c4121515ced"}
Sep 17 16:55:43.266653 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:43.266579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" event={"ID":"47703c88-87a0-4e64-94f9-2306d99465c5","Type":"ContainerStarted","Data":"3d37c6a4022e8ab561eaf446735169ffdc5a86ecfdb7f8c6763b55ec8de99fac"}
Sep 17 16:55:43.266834 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:43.266810    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:43.266881 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:43.266869    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:43.267993 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:43.267970    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 16:55:43.282580 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:43.282058    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" podStartSLOduration=3.39183855 podStartE2EDuration="7.282041982s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.890203432s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:55:43.281361872 +0000 UTC m=+808.847473595" watchObservedRunningTime="2026-09-17 16:55:43.282041982 +0000 UTC m=+808.848153706"
Sep 17 16:55:43.299313 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:43.299266    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" podStartSLOduration=3.447959725 podStartE2EDuration="7.29925197s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.851292245s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:55:43.297416044 +0000 UTC m=+808.863527778" watchObservedRunningTime="2026-09-17 16:55:43.29925197 +0000 UTC m=+808.865363691"
Sep 17 16:55:44.272651 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:44.272591    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 16:55:44.273210 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:44.273189    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:44.278316 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:44.273440    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 16:55:45.276066 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:45.276019    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 16:55:48.155187 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:48.155145    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 16:55:48.155571 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:48.155145    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 16:55:49.276944 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:49.276864    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:55:49.277448 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:49.277422    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 16:55:50.280895 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:50.280865    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:55:50.281581 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:50.281551    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 16:55:58.155493 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:58.155447    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 16:55:58.156009 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:58.155464    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 16:55:59.277838 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:55:59.277802    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 16:56:00.281466 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:00.281430    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 16:56:08.154967 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:08.154906    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 16:56:08.155685 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:08.154952    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 16:56:09.277852 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:09.277814    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 16:56:10.281877 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:10.281835    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 16:56:18.155844 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:18.155799    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 16:56:18.156257 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:18.155798    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 16:56:19.278133 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:19.278097    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 16:56:20.281790 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:20.281754    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 16:56:28.155035 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:28.154987    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 16:56:28.155508 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:28.154987    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 16:56:29.278099 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:29.278053    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 16:56:30.281580 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:30.281542    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 16:56:34.819149 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:34.819097    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 16:56:38.156171 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:38.156142    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:56:39.278185 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:39.278147    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 16:56:40.282002 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:40.281961    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 16:56:44.823712 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:44.823685    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:56:46.217073 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:46.217037    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"]
Sep 17 16:56:46.217530 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:46.217304    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kserve-container" containerID="cri-o://3ec51af7d8e3edda7d3c1cd552d8f0d1926e447119bdf3d2fe4a757c61886653" gracePeriod=30
Sep 17 16:56:46.217530 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:46.217485    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kube-rbac-proxy" containerID="cri-o://465a0538ae60044a453521286e7b892572b0252e785ba41531ee16eae34cf465" gracePeriod=30
Sep 17 16:56:46.236507 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:46.236481    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:56:46.250145 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:46.250120    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"]
Sep 17 16:56:46.250371 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:46.250348    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kserve-container" containerID="cri-o://ce662c7f42b697487532ecb893e5ce09cb5a9738e365c13792a0cad4dafce647" gracePeriod=30
Sep 17 16:56:46.250467 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:46.250386    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kube-rbac-proxy" containerID="cri-o://d9e42961a625200fd448e3ac8bf62650d784978c1fec6a466a7967fb422bd818" gracePeriod=30
Sep 17 16:56:46.337126 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:46.337098    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:56:46.540951 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:46.540850    2565 generic.go:356] "Generic (PLEG): container finished" podID="72bfac89-404c-41e1-9f64-88bee468104f" containerID="465a0538ae60044a453521286e7b892572b0252e785ba41531ee16eae34cf465" exitCode=2
Sep 17 16:56:46.540951 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:46.540935    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" event={"ID":"72bfac89-404c-41e1-9f64-88bee468104f","Type":"ContainerDied","Data":"465a0538ae60044a453521286e7b892572b0252e785ba41531ee16eae34cf465"}
Sep 17 16:56:46.543085 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:46.543064    2565 generic.go:356] "Generic (PLEG): container finished" podID="b2a31cd0-50fd-4891-a27d-e85645918526" containerID="d9e42961a625200fd448e3ac8bf62650d784978c1fec6a466a7967fb422bd818" exitCode=2
Sep 17 16:56:46.543206 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:46.543138    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" event={"ID":"b2a31cd0-50fd-4891-a27d-e85645918526","Type":"ContainerDied","Data":"d9e42961a625200fd448e3ac8bf62650d784978c1fec6a466a7967fb422bd818"}
Sep 17 16:56:48.150261 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:48.150202    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 17 16:56:48.150261 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:48.150223    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 17 16:56:48.155741 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:48.155706    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 16:56:49.277847 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:49.277819    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:56:50.281946 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.281888    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 16:56:50.553701 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.553677    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:56:50.561735 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.561710    2565 generic.go:356] "Generic (PLEG): container finished" podID="72bfac89-404c-41e1-9f64-88bee468104f" containerID="3ec51af7d8e3edda7d3c1cd552d8f0d1926e447119bdf3d2fe4a757c61886653" exitCode=0
Sep 17 16:56:50.561835 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.561770    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"
Sep 17 16:56:50.561835 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.561782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" event={"ID":"72bfac89-404c-41e1-9f64-88bee468104f","Type":"ContainerDied","Data":"3ec51af7d8e3edda7d3c1cd552d8f0d1926e447119bdf3d2fe4a757c61886653"}
Sep 17 16:56:50.561930 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.561842    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t" event={"ID":"72bfac89-404c-41e1-9f64-88bee468104f","Type":"ContainerDied","Data":"8b7b4eb82842cdeda9694fe109a4911872e968dedf233b9b88c56bf8e645a635"}
Sep 17 16:56:50.561930 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.561872    2565 scope.go:117] "RemoveContainer" containerID="465a0538ae60044a453521286e7b892572b0252e785ba41531ee16eae34cf465"
Sep 17 16:56:50.569697 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.569680    2565 scope.go:117] "RemoveContainer" containerID="3ec51af7d8e3edda7d3c1cd552d8f0d1926e447119bdf3d2fe4a757c61886653"
Sep 17 16:56:50.580446 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.580425    2565 scope.go:117] "RemoveContainer" containerID="b5f52050130d90f09b84f0848cdbc3ae968a561924d5d9b5de62421925543617"
Sep 17 16:56:50.588262 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.588247    2565 scope.go:117] "RemoveContainer" containerID="465a0538ae60044a453521286e7b892572b0252e785ba41531ee16eae34cf465"
Sep 17 16:56:50.588536 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:56:50.588512    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"465a0538ae60044a453521286e7b892572b0252e785ba41531ee16eae34cf465\": container with ID starting with 465a0538ae60044a453521286e7b892572b0252e785ba41531ee16eae34cf465 not found: ID does not exist" containerID="465a0538ae60044a453521286e7b892572b0252e785ba41531ee16eae34cf465"
Sep 17 16:56:50.588642 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.588539    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465a0538ae60044a453521286e7b892572b0252e785ba41531ee16eae34cf465"} err="failed to get container status \"465a0538ae60044a453521286e7b892572b0252e785ba41531ee16eae34cf465\": rpc error: code = NotFound desc = could not find container \"465a0538ae60044a453521286e7b892572b0252e785ba41531ee16eae34cf465\": container with ID starting with 465a0538ae60044a453521286e7b892572b0252e785ba41531ee16eae34cf465 not found: ID does not exist"
Sep 17 16:56:50.588642 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.588555    2565 scope.go:117] "RemoveContainer" containerID="3ec51af7d8e3edda7d3c1cd552d8f0d1926e447119bdf3d2fe4a757c61886653"
Sep 17 16:56:50.588805 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:56:50.588789    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ec51af7d8e3edda7d3c1cd552d8f0d1926e447119bdf3d2fe4a757c61886653\": container with ID starting with 3ec51af7d8e3edda7d3c1cd552d8f0d1926e447119bdf3d2fe4a757c61886653 not found: ID does not exist" containerID="3ec51af7d8e3edda7d3c1cd552d8f0d1926e447119bdf3d2fe4a757c61886653"
Sep 17 16:56:50.588868 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.588809    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ec51af7d8e3edda7d3c1cd552d8f0d1926e447119bdf3d2fe4a757c61886653"} err="failed to get container status \"3ec51af7d8e3edda7d3c1cd552d8f0d1926e447119bdf3d2fe4a757c61886653\": rpc error: code = NotFound desc = could not find container \"3ec51af7d8e3edda7d3c1cd552d8f0d1926e447119bdf3d2fe4a757c61886653\": container with ID starting with 3ec51af7d8e3edda7d3c1cd552d8f0d1926e447119bdf3d2fe4a757c61886653 not found: ID does not exist"
Sep 17 16:56:50.588868 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.588821    2565 scope.go:117] "RemoveContainer" containerID="b5f52050130d90f09b84f0848cdbc3ae968a561924d5d9b5de62421925543617"
Sep 17 16:56:50.589086 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:56:50.589070    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f52050130d90f09b84f0848cdbc3ae968a561924d5d9b5de62421925543617\": container with ID starting with b5f52050130d90f09b84f0848cdbc3ae968a561924d5d9b5de62421925543617 not found: ID does not exist" containerID="b5f52050130d90f09b84f0848cdbc3ae968a561924d5d9b5de62421925543617"
Sep 17 16:56:50.589159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.589090    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f52050130d90f09b84f0848cdbc3ae968a561924d5d9b5de62421925543617"} err="failed to get container status \"b5f52050130d90f09b84f0848cdbc3ae968a561924d5d9b5de62421925543617\": rpc error: code = NotFound desc = could not find container \"b5f52050130d90f09b84f0848cdbc3ae968a561924d5d9b5de62421925543617\": container with ID starting with b5f52050130d90f09b84f0848cdbc3ae968a561924d5d9b5de62421925543617 not found: ID does not exist"
Sep 17 16:56:50.665893 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.665873    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grtpn\" (UniqueName: \"kubernetes.io/projected/72bfac89-404c-41e1-9f64-88bee468104f-kube-api-access-grtpn\") pod \"72bfac89-404c-41e1-9f64-88bee468104f\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") "
Sep 17 16:56:50.666005 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.665902    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72bfac89-404c-41e1-9f64-88bee468104f-cabundle-cert\") pod \"72bfac89-404c-41e1-9f64-88bee468104f\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") "
Sep 17 16:56:50.666005 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.665937    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72bfac89-404c-41e1-9f64-88bee468104f-proxy-tls\") pod \"72bfac89-404c-41e1-9f64-88bee468104f\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") "
Sep 17 16:56:50.666121 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.666069    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72bfac89-404c-41e1-9f64-88bee468104f-kserve-provision-location\") pod \"72bfac89-404c-41e1-9f64-88bee468104f\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") "
Sep 17 16:56:50.666173 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.666117    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72bfac89-404c-41e1-9f64-88bee468104f-kube-rbac-proxy-sar-config\") pod \"72bfac89-404c-41e1-9f64-88bee468104f\" (UID: \"72bfac89-404c-41e1-9f64-88bee468104f\") "
Sep 17 16:56:50.666245 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.666221    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72bfac89-404c-41e1-9f64-88bee468104f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "72bfac89-404c-41e1-9f64-88bee468104f" (UID: "72bfac89-404c-41e1-9f64-88bee468104f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:56:50.666395 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.666372    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bfac89-404c-41e1-9f64-88bee468104f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "72bfac89-404c-41e1-9f64-88bee468104f" (UID: "72bfac89-404c-41e1-9f64-88bee468104f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:56:50.666469 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.666424    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72bfac89-404c-41e1-9f64-88bee468104f-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:56:50.666530 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.666502    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72bfac89-404c-41e1-9f64-88bee468104f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "72bfac89-404c-41e1-9f64-88bee468104f" (UID: "72bfac89-404c-41e1-9f64-88bee468104f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:56:50.667874 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.667849    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72bfac89-404c-41e1-9f64-88bee468104f-kube-api-access-grtpn" (OuterVolumeSpecName: "kube-api-access-grtpn") pod "72bfac89-404c-41e1-9f64-88bee468104f" (UID: "72bfac89-404c-41e1-9f64-88bee468104f"). InnerVolumeSpecName "kube-api-access-grtpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:56:50.667994 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.667923    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72bfac89-404c-41e1-9f64-88bee468104f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "72bfac89-404c-41e1-9f64-88bee468104f" (UID: "72bfac89-404c-41e1-9f64-88bee468104f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:56:50.767820 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.767798    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grtpn\" (UniqueName: \"kubernetes.io/projected/72bfac89-404c-41e1-9f64-88bee468104f-kube-api-access-grtpn\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:56:50.767907 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.767823    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72bfac89-404c-41e1-9f64-88bee468104f-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:56:50.767907 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.767834    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72bfac89-404c-41e1-9f64-88bee468104f-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:56:50.767907 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.767842    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72bfac89-404c-41e1-9f64-88bee468104f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:56:50.876408 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.876380    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"]
Sep 17 16:56:50.880084 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:50.880065    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-775495c6b4-np76t"]
Sep 17 16:56:51.394067 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.394047    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:56:51.473441 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.473410    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2a31cd0-50fd-4891-a27d-e85645918526-proxy-tls\") pod \"b2a31cd0-50fd-4891-a27d-e85645918526\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") "
Sep 17 16:56:51.473566 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.473461    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2a31cd0-50fd-4891-a27d-e85645918526-kube-rbac-proxy-sar-config\") pod \"b2a31cd0-50fd-4891-a27d-e85645918526\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") "
Sep 17 16:56:51.473566 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.473492    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49f9m\" (UniqueName: \"kubernetes.io/projected/b2a31cd0-50fd-4891-a27d-e85645918526-kube-api-access-49f9m\") pod \"b2a31cd0-50fd-4891-a27d-e85645918526\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") "
Sep 17 16:56:51.473566 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.473528    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2a31cd0-50fd-4891-a27d-e85645918526-cabundle-cert\") pod \"b2a31cd0-50fd-4891-a27d-e85645918526\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") "
Sep 17 16:56:51.473726 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.473581    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2a31cd0-50fd-4891-a27d-e85645918526-kserve-provision-location\") pod \"b2a31cd0-50fd-4891-a27d-e85645918526\" (UID: \"b2a31cd0-50fd-4891-a27d-e85645918526\") "
Sep 17 16:56:51.473930 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.473888    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a31cd0-50fd-4891-a27d-e85645918526-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b2a31cd0-50fd-4891-a27d-e85645918526" (UID: "b2a31cd0-50fd-4891-a27d-e85645918526"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:56:51.474065 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.473936    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a31cd0-50fd-4891-a27d-e85645918526-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b2a31cd0-50fd-4891-a27d-e85645918526" (UID: "b2a31cd0-50fd-4891-a27d-e85645918526"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:56:51.474065 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.473959    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a31cd0-50fd-4891-a27d-e85645918526-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b2a31cd0-50fd-4891-a27d-e85645918526" (UID: "b2a31cd0-50fd-4891-a27d-e85645918526"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:56:51.475494 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.475471    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a31cd0-50fd-4891-a27d-e85645918526-kube-api-access-49f9m" (OuterVolumeSpecName: "kube-api-access-49f9m") pod "b2a31cd0-50fd-4891-a27d-e85645918526" (UID: "b2a31cd0-50fd-4891-a27d-e85645918526"). InnerVolumeSpecName "kube-api-access-49f9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:56:51.475582 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.475517    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a31cd0-50fd-4891-a27d-e85645918526-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2a31cd0-50fd-4891-a27d-e85645918526" (UID: "b2a31cd0-50fd-4891-a27d-e85645918526"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:56:51.568464 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.568404    2565 generic.go:356] "Generic (PLEG): container finished" podID="b2a31cd0-50fd-4891-a27d-e85645918526" containerID="ce662c7f42b697487532ecb893e5ce09cb5a9738e365c13792a0cad4dafce647" exitCode=0
Sep 17 16:56:51.568464 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.568457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" event={"ID":"b2a31cd0-50fd-4891-a27d-e85645918526","Type":"ContainerDied","Data":"ce662c7f42b697487532ecb893e5ce09cb5a9738e365c13792a0cad4dafce647"}
Sep 17 16:56:51.568597 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.568503    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb" event={"ID":"b2a31cd0-50fd-4891-a27d-e85645918526","Type":"ContainerDied","Data":"6b86aed703d96a2f7f0af8fbb76f936f66d2d7ed259e7d39b12c5c280a0e3e13"}
Sep 17 16:56:51.568597 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.568461    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"
Sep 17 16:56:51.568674 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.568533    2565 scope.go:117] "RemoveContainer" containerID="d9e42961a625200fd448e3ac8bf62650d784978c1fec6a466a7967fb422bd818"
Sep 17 16:56:51.574097 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.574078    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2a31cd0-50fd-4891-a27d-e85645918526-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:56:51.574171 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.574099    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49f9m\" (UniqueName: \"kubernetes.io/projected/b2a31cd0-50fd-4891-a27d-e85645918526-kube-api-access-49f9m\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:56:51.574171 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.574109    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2a31cd0-50fd-4891-a27d-e85645918526-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:56:51.574171 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.574117    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2a31cd0-50fd-4891-a27d-e85645918526-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:56:51.574171 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.574125    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2a31cd0-50fd-4891-a27d-e85645918526-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:56:51.576810 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.576795    2565 scope.go:117] "RemoveContainer" containerID="ce662c7f42b697487532ecb893e5ce09cb5a9738e365c13792a0cad4dafce647"
Sep 17 16:56:51.584424 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.584409    2565 scope.go:117] "RemoveContainer" containerID="19b36a6a8a60020e40f51c3d272eb7e56c9df14952537e1f222f43c070ca2cca"
Sep 17 16:56:51.588153 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.588131    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"]
Sep 17 16:56:51.591873 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.591853    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-67d6bcd4b5-bwhcb"]
Sep 17 16:56:51.592251 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.592233    2565 scope.go:117] "RemoveContainer" containerID="d9e42961a625200fd448e3ac8bf62650d784978c1fec6a466a7967fb422bd818"
Sep 17 16:56:51.592490 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:56:51.592471    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e42961a625200fd448e3ac8bf62650d784978c1fec6a466a7967fb422bd818\": container with ID starting with d9e42961a625200fd448e3ac8bf62650d784978c1fec6a466a7967fb422bd818 not found: ID does not exist" containerID="d9e42961a625200fd448e3ac8bf62650d784978c1fec6a466a7967fb422bd818"
Sep 17 16:56:51.592559 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.592502    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e42961a625200fd448e3ac8bf62650d784978c1fec6a466a7967fb422bd818"} err="failed to get container status \"d9e42961a625200fd448e3ac8bf62650d784978c1fec6a466a7967fb422bd818\": rpc error: code = NotFound desc = could not find container \"d9e42961a625200fd448e3ac8bf62650d784978c1fec6a466a7967fb422bd818\": container with ID starting with d9e42961a625200fd448e3ac8bf62650d784978c1fec6a466a7967fb422bd818 not found: ID does not exist"
Sep 17 16:56:51.592559 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.592525    2565 scope.go:117] "RemoveContainer" containerID="ce662c7f42b697487532ecb893e5ce09cb5a9738e365c13792a0cad4dafce647"
Sep 17 16:56:51.592768 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:56:51.592751    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce662c7f42b697487532ecb893e5ce09cb5a9738e365c13792a0cad4dafce647\": container with ID starting with ce662c7f42b697487532ecb893e5ce09cb5a9738e365c13792a0cad4dafce647 not found: ID does not exist" containerID="ce662c7f42b697487532ecb893e5ce09cb5a9738e365c13792a0cad4dafce647"
Sep 17 16:56:51.592807 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.592776    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce662c7f42b697487532ecb893e5ce09cb5a9738e365c13792a0cad4dafce647"} err="failed to get container status \"ce662c7f42b697487532ecb893e5ce09cb5a9738e365c13792a0cad4dafce647\": rpc error: code = NotFound desc = could not find container \"ce662c7f42b697487532ecb893e5ce09cb5a9738e365c13792a0cad4dafce647\": container with ID starting with ce662c7f42b697487532ecb893e5ce09cb5a9738e365c13792a0cad4dafce647 not found: ID does not exist"
Sep 17 16:56:51.592807 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.592795    2565 scope.go:117] "RemoveContainer" containerID="19b36a6a8a60020e40f51c3d272eb7e56c9df14952537e1f222f43c070ca2cca"
Sep 17 16:56:51.593079 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:56:51.593064    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b36a6a8a60020e40f51c3d272eb7e56c9df14952537e1f222f43c070ca2cca\": container with ID starting with 19b36a6a8a60020e40f51c3d272eb7e56c9df14952537e1f222f43c070ca2cca not found: ID does not exist" containerID="19b36a6a8a60020e40f51c3d272eb7e56c9df14952537e1f222f43c070ca2cca"
Sep 17 16:56:51.593141 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:51.593083    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b36a6a8a60020e40f51c3d272eb7e56c9df14952537e1f222f43c070ca2cca"} err="failed to get container status \"19b36a6a8a60020e40f51c3d272eb7e56c9df14952537e1f222f43c070ca2cca\": rpc error: code = NotFound desc = could not find container \"19b36a6a8a60020e40f51c3d272eb7e56c9df14952537e1f222f43c070ca2cca\": container with ID starting with 19b36a6a8a60020e40f51c3d272eb7e56c9df14952537e1f222f43c070ca2cca not found: ID does not exist"
Sep 17 16:56:52.823346 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:52.823318    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72bfac89-404c-41e1-9f64-88bee468104f" path="/var/lib/kubelet/pods/72bfac89-404c-41e1-9f64-88bee468104f/volumes"
Sep 17 16:56:52.823781 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:52.823768    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" path="/var/lib/kubelet/pods/b2a31cd0-50fd-4891-a27d-e85645918526/volumes"
Sep 17 16:56:56.441125 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.441087    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:56:56.469997 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.469966    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"]
Sep 17 16:56:56.470339 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470322    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kube-rbac-proxy"
Sep 17 16:56:56.470388 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470342    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kube-rbac-proxy"
Sep 17 16:56:56.470388 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470353    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kserve-container"
Sep 17 16:56:56.470388 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470359    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kserve-container"
Sep 17 16:56:56.470388 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470365    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="storage-initializer"
Sep 17 16:56:56.470388 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470370    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="storage-initializer"
Sep 17 16:56:56.470388 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470380    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="storage-initializer"
Sep 17 16:56:56.470388 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470386    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="storage-initializer"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470392    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kube-rbac-proxy"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470397    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kube-rbac-proxy"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470414    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kube-rbac-proxy"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470420    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kube-rbac-proxy"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470426    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kserve-container"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470430    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kserve-container"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470440    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="storage-initializer"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470445    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="storage-initializer"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470451    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kserve-container"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470456    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kserve-container"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470461    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kserve-container"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470465    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kserve-container"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470478    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="storage-initializer"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470485    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="storage-initializer"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470496    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kube-rbac-proxy"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470502    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kube-rbac-proxy"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470550    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kserve-container"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470557    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="72bfac89-404c-41e1-9f64-88bee468104f" containerName="kube-rbac-proxy"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470565    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kube-rbac-proxy"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470574    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kserve-container"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470582    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kube-rbac-proxy"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470587    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71333a7b-dba5-4488-8dfc-930854c8dfe3" containerName="kserve-container"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470593    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2a31cd0-50fd-4891-a27d-e85645918526" containerName="kserve-container"
Sep 17 16:56:56.470591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.470599    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f52e550-2266-464e-bbc1-b12e1b7f3f55" containerName="kube-rbac-proxy"
Sep 17 16:56:56.475655 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.475636    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:56.478273 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.478253    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-42374c-predictor-serving-cert\""
Sep 17 16:56:56.478406 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.478390    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-42374c-kube-rbac-proxy-sar-config\""
Sep 17 16:56:56.484887 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.484866    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"]
Sep 17 16:56:56.609087 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.609050    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wbzh\" (UniqueName: \"kubernetes.io/projected/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kube-api-access-2wbzh\") pod \"isvc-primary-42374c-predictor-798b4f4b89-bb7vt\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") " pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:56.609188 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.609096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kserve-provision-location\") pod \"isvc-primary-42374c-predictor-798b4f4b89-bb7vt\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") " pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:56.609188 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.609139    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d11abf8-027d-4f87-8a06-e57af2fde6bf-cabundle-cert\") pod \"isvc-primary-42374c-predictor-798b4f4b89-bb7vt\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") " pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:56.609188 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.609165    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kube-rbac-proxy-sar-config\") pod \"isvc-primary-42374c-predictor-798b4f4b89-bb7vt\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") " pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:56.609316 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.609255    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d11abf8-027d-4f87-8a06-e57af2fde6bf-proxy-tls\") pod \"isvc-primary-42374c-predictor-798b4f4b89-bb7vt\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") " pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:56.710191 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.710112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d11abf8-027d-4f87-8a06-e57af2fde6bf-proxy-tls\") pod \"isvc-primary-42374c-predictor-798b4f4b89-bb7vt\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") " pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:56.710191 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.710168    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wbzh\" (UniqueName: \"kubernetes.io/projected/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kube-api-access-2wbzh\") pod \"isvc-primary-42374c-predictor-798b4f4b89-bb7vt\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") " pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:56.710358 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:56:56.710275    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-42374c-predictor-serving-cert: secret "isvc-primary-42374c-predictor-serving-cert" not found
Sep 17 16:56:56.710358 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:56:56.710330    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d11abf8-027d-4f87-8a06-e57af2fde6bf-proxy-tls podName:4d11abf8-027d-4f87-8a06-e57af2fde6bf nodeName:}" failed. No retries permitted until 2026-09-17 16:56:57.210311513 +0000 UTC m=+882.776423228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4d11abf8-027d-4f87-8a06-e57af2fde6bf-proxy-tls") pod "isvc-primary-42374c-predictor-798b4f4b89-bb7vt" (UID: "4d11abf8-027d-4f87-8a06-e57af2fde6bf") : secret "isvc-primary-42374c-predictor-serving-cert" not found
Sep 17 16:56:56.710482 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.710353    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kserve-provision-location\") pod \"isvc-primary-42374c-predictor-798b4f4b89-bb7vt\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") " pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:56.710482 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.710385    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d11abf8-027d-4f87-8a06-e57af2fde6bf-cabundle-cert\") pod \"isvc-primary-42374c-predictor-798b4f4b89-bb7vt\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") " pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:56.710482 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.710420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kube-rbac-proxy-sar-config\") pod \"isvc-primary-42374c-predictor-798b4f4b89-bb7vt\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") " pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:56.710763 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.710743    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kserve-provision-location\") pod \"isvc-primary-42374c-predictor-798b4f4b89-bb7vt\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") " pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:56.711011 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.710997    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d11abf8-027d-4f87-8a06-e57af2fde6bf-cabundle-cert\") pod \"isvc-primary-42374c-predictor-798b4f4b89-bb7vt\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") " pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:56.711065 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.711049    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kube-rbac-proxy-sar-config\") pod \"isvc-primary-42374c-predictor-798b4f4b89-bb7vt\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") " pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:56.719635 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.719615    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wbzh\" (UniqueName: \"kubernetes.io/projected/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kube-api-access-2wbzh\") pod \"isvc-primary-42374c-predictor-798b4f4b89-bb7vt\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") " pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:56.951198 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:56.951172    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:56:57.214468 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:57.214444    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d11abf8-027d-4f87-8a06-e57af2fde6bf-proxy-tls\") pod \"isvc-primary-42374c-predictor-798b4f4b89-bb7vt\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") " pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:57.216720 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:57.216694    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d11abf8-027d-4f87-8a06-e57af2fde6bf-proxy-tls\") pod \"isvc-primary-42374c-predictor-798b4f4b89-bb7vt\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") " pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:57.406145 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:57.406114    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:56:57.676937 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:57.676902    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"]
Sep 17 16:56:57.678761 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:56:57.678737    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d11abf8_027d_4f87_8a06_e57af2fde6bf.slice/crio-1d7e39f737f50e716abefeb6247538c2da2ff43317576c6ef28f02144319f7c1 WatchSource:0}: Error finding container 1d7e39f737f50e716abefeb6247538c2da2ff43317576c6ef28f02144319f7c1: Status 404 returned error can't find the container with id 1d7e39f737f50e716abefeb6247538c2da2ff43317576c6ef28f02144319f7c1
Sep 17 16:56:58.598906 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:58.598823    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" event={"ID":"4d11abf8-027d-4f87-8a06-e57af2fde6bf","Type":"ContainerStarted","Data":"d3f036ccbc71a3b6e7213488f6376f929bcea30a98455d3b7338dbdb629f194d"}
Sep 17 16:56:58.598906 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:56:58.598897    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" event={"ID":"4d11abf8-027d-4f87-8a06-e57af2fde6bf","Type":"ContainerStarted","Data":"1d7e39f737f50e716abefeb6247538c2da2ff43317576c6ef28f02144319f7c1"}
Sep 17 16:57:00.282110 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:00.282076    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:57:02.616136 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:02.616098    2565 generic.go:356] "Generic (PLEG): container finished" podID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerID="d3f036ccbc71a3b6e7213488f6376f929bcea30a98455d3b7338dbdb629f194d" exitCode=0
Sep 17 16:57:02.616515 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:02.616177    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" event={"ID":"4d11abf8-027d-4f87-8a06-e57af2fde6bf","Type":"ContainerDied","Data":"d3f036ccbc71a3b6e7213488f6376f929bcea30a98455d3b7338dbdb629f194d"}
Sep 17 16:57:03.622592 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:03.622551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" event={"ID":"4d11abf8-027d-4f87-8a06-e57af2fde6bf","Type":"ContainerStarted","Data":"2d60c713ba68ed154dcc7d00043ba0e0cfb60c8be63cc93a4f5f444aebbd1c4e"}
Sep 17 16:57:03.623081 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:03.622622    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" event={"ID":"4d11abf8-027d-4f87-8a06-e57af2fde6bf","Type":"ContainerStarted","Data":"a81de15aaee276aa33151beba04b84816add58366b3cf77f3d4f5a17d564c005"}
Sep 17 16:57:03.623081 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:03.622844    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:57:03.640724 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:03.640680    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" podStartSLOduration=3.577606699 podStartE2EDuration="7.640665736s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.063059037s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:57:03.639560159 +0000 UTC m=+889.205671882" watchObservedRunningTime="2026-09-17 16:57:03.640665736 +0000 UTC m=+889.206777458"
Sep 17 16:57:04.627014 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:04.626984    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:57:04.628441 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:04.628409    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 16:57:05.631113 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:05.631068    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 16:57:10.635565 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:10.635535    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:57:10.636060 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:10.635986    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 16:57:16.428333 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:16.428305    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:57:16.766232 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:16.766157    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:57:20.636684 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:20.636646    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 16:57:26.494840 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.494804    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:57:26.573369 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.573334    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"]
Sep 17 16:57:26.573637 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.573579    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kserve-container" containerID="cri-o://b3100e5ade74b07bd241956658b23c2ba4caa6c1c1c250744d0f2ad1e4939e22" gracePeriod=30
Sep 17 16:57:26.573758 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.573630    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kube-rbac-proxy" containerID="cri-o://7f5378300a7f42aa930c6b17fe54f7c83d0e08d2fb38f99e282a9bf3c21d9990" gracePeriod=30
Sep 17 16:57:26.597329 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.597296    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:57:26.614343 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.614318    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"]
Sep 17 16:57:26.614591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.614547    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kserve-container" containerID="cri-o://3d37c6a4022e8ab561eaf446735169ffdc5a86ecfdb7f8c6763b55ec8de99fac" gracePeriod=30
Sep 17 16:57:26.614591 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.614578    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kube-rbac-proxy" containerID="cri-o://92350be3353780e857ca5d74c608814dca0bde6e964afba6bed05c4121515ced" gracePeriod=30
Sep 17 16:57:26.690217 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.690185    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:57:26.694328 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.694301    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"]
Sep 17 16:57:26.697582 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.697565    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:57:26.699707 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.699688    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-c61ee-predictor-serving-cert\""
Sep 17 16:57:26.699954 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.699933    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-c61ee-kube-rbac-proxy-sar-config\""
Sep 17 16:57:26.707109 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.707087    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"]
Sep 17 16:57:26.715350 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.715326    2565 generic.go:356] "Generic (PLEG): container finished" podID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerID="7f5378300a7f42aa930c6b17fe54f7c83d0e08d2fb38f99e282a9bf3c21d9990" exitCode=2
Sep 17 16:57:26.715436 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.715374    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" event={"ID":"cb36dbe6-9d3f-418e-bfa9-43ec3862f119","Type":"ContainerDied","Data":"7f5378300a7f42aa930c6b17fe54f7c83d0e08d2fb38f99e282a9bf3c21d9990"}
Sep 17 16:57:26.732848 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.732828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/550bae87-0b56-4165-a460-ff9e4ce671dc-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn\" (UID: \"550bae87-0b56-4165-a460-ff9e4ce671dc\") " pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:57:26.732947 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.732886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/550bae87-0b56-4165-a460-ff9e4ce671dc-proxy-tls\") pod \"message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn\" (UID: \"550bae87-0b56-4165-a460-ff9e4ce671dc\") " pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:57:26.733013 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.732995    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmdst\" (UniqueName: \"kubernetes.io/projected/550bae87-0b56-4165-a460-ff9e4ce671dc-kube-api-access-zmdst\") pod \"message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn\" (UID: \"550bae87-0b56-4165-a460-ff9e4ce671dc\") " pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:57:26.746523 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.746461    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:57:26.833752 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.833728    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/550bae87-0b56-4165-a460-ff9e4ce671dc-proxy-tls\") pod \"message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn\" (UID: \"550bae87-0b56-4165-a460-ff9e4ce671dc\") " pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:57:26.833851 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.833799    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmdst\" (UniqueName: \"kubernetes.io/projected/550bae87-0b56-4165-a460-ff9e4ce671dc-kube-api-access-zmdst\") pod \"message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn\" (UID: \"550bae87-0b56-4165-a460-ff9e4ce671dc\") " pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:57:26.833934 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.833892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/550bae87-0b56-4165-a460-ff9e4ce671dc-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn\" (UID: \"550bae87-0b56-4165-a460-ff9e4ce671dc\") " pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:57:26.834002 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:57:26.833979    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-serving-cert: secret "message-dumper-raw-c61ee-predictor-serving-cert" not found
Sep 17 16:57:26.834063 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:57:26.834056    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/550bae87-0b56-4165-a460-ff9e4ce671dc-proxy-tls podName:550bae87-0b56-4165-a460-ff9e4ce671dc nodeName:}" failed. No retries permitted until 2026-09-17 16:57:27.334035757 +0000 UTC m=+912.900147466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/550bae87-0b56-4165-a460-ff9e4ce671dc-proxy-tls") pod "message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn" (UID: "550bae87-0b56-4165-a460-ff9e4ce671dc") : secret "message-dumper-raw-c61ee-predictor-serving-cert" not found
Sep 17 16:57:26.834616 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.834598    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/550bae87-0b56-4165-a460-ff9e4ce671dc-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn\" (UID: \"550bae87-0b56-4165-a460-ff9e4ce671dc\") " pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:57:26.842225 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:26.842202    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmdst\" (UniqueName: \"kubernetes.io/projected/550bae87-0b56-4165-a460-ff9e4ce671dc-kube-api-access-zmdst\") pod \"message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn\" (UID: \"550bae87-0b56-4165-a460-ff9e4ce671dc\") " pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:57:27.036459 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:27.036390    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:57:27.338004 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:27.337931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/550bae87-0b56-4165-a460-ff9e4ce671dc-proxy-tls\") pod \"message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn\" (UID: \"550bae87-0b56-4165-a460-ff9e4ce671dc\") " pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:57:27.340139 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:27.340121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/550bae87-0b56-4165-a460-ff9e4ce671dc-proxy-tls\") pod \"message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn\" (UID: \"550bae87-0b56-4165-a460-ff9e4ce671dc\") " pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:57:27.608579 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:27.608496    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:57:27.721822 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:27.721792    2565 generic.go:356] "Generic (PLEG): container finished" podID="47703c88-87a0-4e64-94f9-2306d99465c5" containerID="92350be3353780e857ca5d74c608814dca0bde6e964afba6bed05c4121515ced" exitCode=2
Sep 17 16:57:27.721970 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:27.721827    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" event={"ID":"47703c88-87a0-4e64-94f9-2306d99465c5","Type":"ContainerDied","Data":"92350be3353780e857ca5d74c608814dca0bde6e964afba6bed05c4121515ced"}
Sep 17 16:57:27.878966 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:27.878942    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"]
Sep 17 16:57:27.880850 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:57:27.880822    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550bae87_0b56_4165_a460_ff9e4ce671dc.slice/crio-9e8080a803d12a4280e013949b60bae82324f29d01c1d6f65947b9d00548105a WatchSource:0}: Error finding container 9e8080a803d12a4280e013949b60bae82324f29d01c1d6f65947b9d00548105a: Status 404 returned error can't find the container with id 9e8080a803d12a4280e013949b60bae82324f29d01c1d6f65947b9d00548105a
Sep 17 16:57:28.728505 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:28.728464    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn" event={"ID":"550bae87-0b56-4165-a460-ff9e4ce671dc","Type":"ContainerStarted","Data":"9e8080a803d12a4280e013949b60bae82324f29d01c1d6f65947b9d00548105a"}
Sep 17 16:57:29.272290 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:29.272261    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 16:57:29.277436 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:29.277410    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 16:57:29.734369 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:29.734334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn" event={"ID":"550bae87-0b56-4165-a460-ff9e4ce671dc","Type":"ContainerStarted","Data":"07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de"}
Sep 17 16:57:29.734736 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:29.734427    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn" event={"ID":"550bae87-0b56-4165-a460-ff9e4ce671dc","Type":"ContainerStarted","Data":"45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432"}
Sep 17 16:57:29.734736 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:29.734455    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:57:29.734736 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:29.734500    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:57:29.736079 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:29.736063    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:57:29.753216 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:29.753166    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn" podStartSLOduration=2.45796861 podStartE2EDuration="3.753150272s" totalImagesPullingTime="1.295181662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:57:29.750072666 +0000 UTC m=+915.316184388" watchObservedRunningTime="2026-09-17 16:57:29.753150272 +0000 UTC m=+915.319261999"
Sep 17 16:57:30.277015 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.276980    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 17 16:57:30.281495 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.281469    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 16:57:30.452862 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.452836    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:57:30.564111 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.564046    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47703c88-87a0-4e64-94f9-2306d99465c5-proxy-tls\") pod \"47703c88-87a0-4e64-94f9-2306d99465c5\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") "
Sep 17 16:57:30.564111 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.564093    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47703c88-87a0-4e64-94f9-2306d99465c5-cabundle-cert\") pod \"47703c88-87a0-4e64-94f9-2306d99465c5\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") "
Sep 17 16:57:30.564301 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.564148    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47703c88-87a0-4e64-94f9-2306d99465c5-kserve-provision-location\") pod \"47703c88-87a0-4e64-94f9-2306d99465c5\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") "
Sep 17 16:57:30.564301 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.564178    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsff4\" (UniqueName: \"kubernetes.io/projected/47703c88-87a0-4e64-94f9-2306d99465c5-kube-api-access-gsff4\") pod \"47703c88-87a0-4e64-94f9-2306d99465c5\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") "
Sep 17 16:57:30.564301 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.564194    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47703c88-87a0-4e64-94f9-2306d99465c5-kube-rbac-proxy-sar-config\") pod \"47703c88-87a0-4e64-94f9-2306d99465c5\" (UID: \"47703c88-87a0-4e64-94f9-2306d99465c5\") "
Sep 17 16:57:30.564526 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.564488    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47703c88-87a0-4e64-94f9-2306d99465c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "47703c88-87a0-4e64-94f9-2306d99465c5" (UID: "47703c88-87a0-4e64-94f9-2306d99465c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:57:30.564586 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.564537    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47703c88-87a0-4e64-94f9-2306d99465c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "47703c88-87a0-4e64-94f9-2306d99465c5" (UID: "47703c88-87a0-4e64-94f9-2306d99465c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:57:30.564643 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.564630    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47703c88-87a0-4e64-94f9-2306d99465c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "47703c88-87a0-4e64-94f9-2306d99465c5" (UID: "47703c88-87a0-4e64-94f9-2306d99465c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:57:30.566167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.566144    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47703c88-87a0-4e64-94f9-2306d99465c5-kube-api-access-gsff4" (OuterVolumeSpecName: "kube-api-access-gsff4") pod "47703c88-87a0-4e64-94f9-2306d99465c5" (UID: "47703c88-87a0-4e64-94f9-2306d99465c5"). InnerVolumeSpecName "kube-api-access-gsff4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:57:30.566387 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.566195    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47703c88-87a0-4e64-94f9-2306d99465c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "47703c88-87a0-4e64-94f9-2306d99465c5" (UID: "47703c88-87a0-4e64-94f9-2306d99465c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:57:30.636277 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.636248    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 16:57:30.665559 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.665536    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47703c88-87a0-4e64-94f9-2306d99465c5-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:57:30.665559 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.665556    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsff4\" (UniqueName: \"kubernetes.io/projected/47703c88-87a0-4e64-94f9-2306d99465c5-kube-api-access-gsff4\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:57:30.665694 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.665565    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47703c88-87a0-4e64-94f9-2306d99465c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:57:30.665694 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.665574    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47703c88-87a0-4e64-94f9-2306d99465c5-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:57:30.665694 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.665583    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47703c88-87a0-4e64-94f9-2306d99465c5-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:57:30.740545 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.740521    2565 generic.go:356] "Generic (PLEG): container finished" podID="47703c88-87a0-4e64-94f9-2306d99465c5" containerID="3d37c6a4022e8ab561eaf446735169ffdc5a86ecfdb7f8c6763b55ec8de99fac" exitCode=0
Sep 17 16:57:30.740951 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.740589    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"
Sep 17 16:57:30.740951 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.740606    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" event={"ID":"47703c88-87a0-4e64-94f9-2306d99465c5","Type":"ContainerDied","Data":"3d37c6a4022e8ab561eaf446735169ffdc5a86ecfdb7f8c6763b55ec8de99fac"}
Sep 17 16:57:30.740951 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.740664    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh" event={"ID":"47703c88-87a0-4e64-94f9-2306d99465c5","Type":"ContainerDied","Data":"3953c5fdfb719a789384de28f8c0a7d5ebbe30e62937cd7c9e0169991046ebc3"}
Sep 17 16:57:30.740951 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.740690    2565 scope.go:117] "RemoveContainer" containerID="92350be3353780e857ca5d74c608814dca0bde6e964afba6bed05c4121515ced"
Sep 17 16:57:30.750223 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.750208    2565 scope.go:117] "RemoveContainer" containerID="3d37c6a4022e8ab561eaf446735169ffdc5a86ecfdb7f8c6763b55ec8de99fac"
Sep 17 16:57:30.758101 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.758086    2565 scope.go:117] "RemoveContainer" containerID="ddea334e743dc15d796a1c9d4f9c257bd4fff2699fdb0c3db6c1db27615308b2"
Sep 17 16:57:30.762820 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.762801    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"]
Sep 17 16:57:30.766121 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.766105    2565 scope.go:117] "RemoveContainer" containerID="92350be3353780e857ca5d74c608814dca0bde6e964afba6bed05c4121515ced"
Sep 17 16:57:30.766422 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:57:30.766399    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92350be3353780e857ca5d74c608814dca0bde6e964afba6bed05c4121515ced\": container with ID starting with 92350be3353780e857ca5d74c608814dca0bde6e964afba6bed05c4121515ced not found: ID does not exist" containerID="92350be3353780e857ca5d74c608814dca0bde6e964afba6bed05c4121515ced"
Sep 17 16:57:30.766474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.766430    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92350be3353780e857ca5d74c608814dca0bde6e964afba6bed05c4121515ced"} err="failed to get container status \"92350be3353780e857ca5d74c608814dca0bde6e964afba6bed05c4121515ced\": rpc error: code = NotFound desc = could not find container \"92350be3353780e857ca5d74c608814dca0bde6e964afba6bed05c4121515ced\": container with ID starting with 92350be3353780e857ca5d74c608814dca0bde6e964afba6bed05c4121515ced not found: ID does not exist"
Sep 17 16:57:30.766474 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.766445    2565 scope.go:117] "RemoveContainer" containerID="3d37c6a4022e8ab561eaf446735169ffdc5a86ecfdb7f8c6763b55ec8de99fac"
Sep 17 16:57:30.766754 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:57:30.766735    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d37c6a4022e8ab561eaf446735169ffdc5a86ecfdb7f8c6763b55ec8de99fac\": container with ID starting with 3d37c6a4022e8ab561eaf446735169ffdc5a86ecfdb7f8c6763b55ec8de99fac not found: ID does not exist" containerID="3d37c6a4022e8ab561eaf446735169ffdc5a86ecfdb7f8c6763b55ec8de99fac"
Sep 17 16:57:30.766820 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.766765    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d37c6a4022e8ab561eaf446735169ffdc5a86ecfdb7f8c6763b55ec8de99fac"} err="failed to get container status \"3d37c6a4022e8ab561eaf446735169ffdc5a86ecfdb7f8c6763b55ec8de99fac\": rpc error: code = NotFound desc = could not find container \"3d37c6a4022e8ab561eaf446735169ffdc5a86ecfdb7f8c6763b55ec8de99fac\": container with ID starting with 3d37c6a4022e8ab561eaf446735169ffdc5a86ecfdb7f8c6763b55ec8de99fac not found: ID does not exist"
Sep 17 16:57:30.766820 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.766783    2565 scope.go:117] "RemoveContainer" containerID="ddea334e743dc15d796a1c9d4f9c257bd4fff2699fdb0c3db6c1db27615308b2"
Sep 17 16:57:30.767106 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:57:30.767084    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddea334e743dc15d796a1c9d4f9c257bd4fff2699fdb0c3db6c1db27615308b2\": container with ID starting with ddea334e743dc15d796a1c9d4f9c257bd4fff2699fdb0c3db6c1db27615308b2 not found: ID does not exist" containerID="ddea334e743dc15d796a1c9d4f9c257bd4fff2699fdb0c3db6c1db27615308b2"
Sep 17 16:57:30.767161 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.767115    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddea334e743dc15d796a1c9d4f9c257bd4fff2699fdb0c3db6c1db27615308b2"} err="failed to get container status \"ddea334e743dc15d796a1c9d4f9c257bd4fff2699fdb0c3db6c1db27615308b2\": rpc error: code = NotFound desc = could not find container \"ddea334e743dc15d796a1c9d4f9c257bd4fff2699fdb0c3db6c1db27615308b2\": container with ID starting with ddea334e743dc15d796a1c9d4f9c257bd4fff2699fdb0c3db6c1db27615308b2 not found: ID does not exist"
Sep 17 16:57:30.767209 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.767197    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-48552-predictor-5cb6c4b997-d87rh"]
Sep 17 16:57:30.823907 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:30.823855    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" path="/var/lib/kubelet/pods/47703c88-87a0-4e64-94f9-2306d99465c5/volumes"
Sep 17 16:57:31.105384 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.105360    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:57:31.168770 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.168744    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-proxy-tls\") pod \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") "
Sep 17 16:57:31.168896 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.168782    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-cabundle-cert\") pod \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") "
Sep 17 16:57:31.168896 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.168833    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kserve-provision-location\") pod \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") "
Sep 17 16:57:31.168896 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.168872    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkgd2\" (UniqueName: \"kubernetes.io/projected/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kube-api-access-vkgd2\") pod \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") "
Sep 17 16:57:31.169024 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.168934    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kube-rbac-proxy-sar-config\") pod \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\" (UID: \"cb36dbe6-9d3f-418e-bfa9-43ec3862f119\") "
Sep 17 16:57:31.169180 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.169147    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cb36dbe6-9d3f-418e-bfa9-43ec3862f119" (UID: "cb36dbe6-9d3f-418e-bfa9-43ec3862f119"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:57:31.169278 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.169154    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cb36dbe6-9d3f-418e-bfa9-43ec3862f119" (UID: "cb36dbe6-9d3f-418e-bfa9-43ec3862f119"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:57:31.169335 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.169312    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb36dbe6-9d3f-418e-bfa9-43ec3862f119" (UID: "cb36dbe6-9d3f-418e-bfa9-43ec3862f119"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:57:31.170849 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.170829    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kube-api-access-vkgd2" (OuterVolumeSpecName: "kube-api-access-vkgd2") pod "cb36dbe6-9d3f-418e-bfa9-43ec3862f119" (UID: "cb36dbe6-9d3f-418e-bfa9-43ec3862f119"). InnerVolumeSpecName "kube-api-access-vkgd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:57:31.170977 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.170864    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb36dbe6-9d3f-418e-bfa9-43ec3862f119" (UID: "cb36dbe6-9d3f-418e-bfa9-43ec3862f119"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:57:31.270088 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.270067    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:57:31.270088 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.270089    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:57:31.270213 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.270098    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:57:31.270213 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.270109    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkgd2\" (UniqueName: \"kubernetes.io/projected/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kube-api-access-vkgd2\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:57:31.270213 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.270118    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb36dbe6-9d3f-418e-bfa9-43ec3862f119-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:57:31.747266 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.747240    2565 generic.go:356] "Generic (PLEG): container finished" podID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerID="b3100e5ade74b07bd241956658b23c2ba4caa6c1c1c250744d0f2ad1e4939e22" exitCode=0
Sep 17 16:57:31.747634 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.747318    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"
Sep 17 16:57:31.747634 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.747334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" event={"ID":"cb36dbe6-9d3f-418e-bfa9-43ec3862f119","Type":"ContainerDied","Data":"b3100e5ade74b07bd241956658b23c2ba4caa6c1c1c250744d0f2ad1e4939e22"}
Sep 17 16:57:31.747634 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.747408    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5" event={"ID":"cb36dbe6-9d3f-418e-bfa9-43ec3862f119","Type":"ContainerDied","Data":"ef4723be7cab4b867466ea7ecc905b7ae5b26606151801f4acb9aa388f0c9ede"}
Sep 17 16:57:31.747634 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.747441    2565 scope.go:117] "RemoveContainer" containerID="7f5378300a7f42aa930c6b17fe54f7c83d0e08d2fb38f99e282a9bf3c21d9990"
Sep 17 16:57:31.755437 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.755420    2565 scope.go:117] "RemoveContainer" containerID="b3100e5ade74b07bd241956658b23c2ba4caa6c1c1c250744d0f2ad1e4939e22"
Sep 17 16:57:31.762550 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.762532    2565 scope.go:117] "RemoveContainer" containerID="dac0382c5aec9bbe8fcfcfb0111b08096b614e938cca1400e7326a8b83e43916"
Sep 17 16:57:31.769272 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.769226    2565 scope.go:117] "RemoveContainer" containerID="7f5378300a7f42aa930c6b17fe54f7c83d0e08d2fb38f99e282a9bf3c21d9990"
Sep 17 16:57:31.769960 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:57:31.769801    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5378300a7f42aa930c6b17fe54f7c83d0e08d2fb38f99e282a9bf3c21d9990\": container with ID starting with 7f5378300a7f42aa930c6b17fe54f7c83d0e08d2fb38f99e282a9bf3c21d9990 not found: ID does not exist" containerID="7f5378300a7f42aa930c6b17fe54f7c83d0e08d2fb38f99e282a9bf3c21d9990"
Sep 17 16:57:31.769960 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.769835    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5378300a7f42aa930c6b17fe54f7c83d0e08d2fb38f99e282a9bf3c21d9990"} err="failed to get container status \"7f5378300a7f42aa930c6b17fe54f7c83d0e08d2fb38f99e282a9bf3c21d9990\": rpc error: code = NotFound desc = could not find container \"7f5378300a7f42aa930c6b17fe54f7c83d0e08d2fb38f99e282a9bf3c21d9990\": container with ID starting with 7f5378300a7f42aa930c6b17fe54f7c83d0e08d2fb38f99e282a9bf3c21d9990 not found: ID does not exist"
Sep 17 16:57:31.769960 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.769859    2565 scope.go:117] "RemoveContainer" containerID="b3100e5ade74b07bd241956658b23c2ba4caa6c1c1c250744d0f2ad1e4939e22"
Sep 17 16:57:31.770176 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:57:31.770149    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3100e5ade74b07bd241956658b23c2ba4caa6c1c1c250744d0f2ad1e4939e22\": container with ID starting with b3100e5ade74b07bd241956658b23c2ba4caa6c1c1c250744d0f2ad1e4939e22 not found: ID does not exist" containerID="b3100e5ade74b07bd241956658b23c2ba4caa6c1c1c250744d0f2ad1e4939e22"
Sep 17 16:57:31.770176 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.770168    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3100e5ade74b07bd241956658b23c2ba4caa6c1c1c250744d0f2ad1e4939e22"} err="failed to get container status \"b3100e5ade74b07bd241956658b23c2ba4caa6c1c1c250744d0f2ad1e4939e22\": rpc error: code = NotFound desc = could not find container \"b3100e5ade74b07bd241956658b23c2ba4caa6c1c1c250744d0f2ad1e4939e22\": container with ID starting with b3100e5ade74b07bd241956658b23c2ba4caa6c1c1c250744d0f2ad1e4939e22 not found: ID does not exist"
Sep 17 16:57:31.770257 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.770184    2565 scope.go:117] "RemoveContainer" containerID="dac0382c5aec9bbe8fcfcfb0111b08096b614e938cca1400e7326a8b83e43916"
Sep 17 16:57:31.770527 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:57:31.770504    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac0382c5aec9bbe8fcfcfb0111b08096b614e938cca1400e7326a8b83e43916\": container with ID starting with dac0382c5aec9bbe8fcfcfb0111b08096b614e938cca1400e7326a8b83e43916 not found: ID does not exist" containerID="dac0382c5aec9bbe8fcfcfb0111b08096b614e938cca1400e7326a8b83e43916"
Sep 17 16:57:31.770592 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.770534    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac0382c5aec9bbe8fcfcfb0111b08096b614e938cca1400e7326a8b83e43916"} err="failed to get container status \"dac0382c5aec9bbe8fcfcfb0111b08096b614e938cca1400e7326a8b83e43916\": rpc error: code = NotFound desc = could not find container \"dac0382c5aec9bbe8fcfcfb0111b08096b614e938cca1400e7326a8b83e43916\": container with ID starting with dac0382c5aec9bbe8fcfcfb0111b08096b614e938cca1400e7326a8b83e43916 not found: ID does not exist"
Sep 17 16:57:31.771348 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.771331    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"]
Sep 17 16:57:31.774740 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:31.774720    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-48552-predictor-578789fd6d-thqm5"]
Sep 17 16:57:32.822866 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:32.822820    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" path="/var/lib/kubelet/pods/cb36dbe6-9d3f-418e-bfa9-43ec3862f119/volumes"
Sep 17 16:57:36.753570 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:36.753541    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:57:40.636108 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:40.636071    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 16:57:46.661568 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.661535    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:57:46.702603 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.702567    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"]
Sep 17 16:57:46.702975 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.702959    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="storage-initializer"
Sep 17 16:57:46.703073 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.702977    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="storage-initializer"
Sep 17 16:57:46.703073 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.702994    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kube-rbac-proxy"
Sep 17 16:57:46.703073 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.703002    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kube-rbac-proxy"
Sep 17 16:57:46.703073 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.703013    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kserve-container"
Sep 17 16:57:46.703073 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.703021    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kserve-container"
Sep 17 16:57:46.703073 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.703038    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="storage-initializer"
Sep 17 16:57:46.703073 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.703047    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="storage-initializer"
Sep 17 16:57:46.703073 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.703056    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kube-rbac-proxy"
Sep 17 16:57:46.703073 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.703063    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kube-rbac-proxy"
Sep 17 16:57:46.703073 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.703075    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kserve-container"
Sep 17 16:57:46.703564 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.703083    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kserve-container"
Sep 17 16:57:46.703564 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.703169    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kube-rbac-proxy"
Sep 17 16:57:46.703564 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.703183    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb36dbe6-9d3f-418e-bfa9-43ec3862f119" containerName="kserve-container"
Sep 17 16:57:46.703564 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.703192    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kube-rbac-proxy"
Sep 17 16:57:46.703564 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.703201    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="47703c88-87a0-4e64-94f9-2306d99465c5" containerName="kserve-container"
Sep 17 16:57:46.707880 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.707861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:46.710442 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.710419    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-c61ee-predictor-serving-cert\""
Sep 17 16:57:46.710549 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.710426    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-c61ee-kube-rbac-proxy-sar-config\""
Sep 17 16:57:46.716778 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.716751    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"]
Sep 17 16:57:46.774135 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.774114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:46.774266 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.774145    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kserve-provision-location\") pod \"isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:46.774266 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.774232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-cabundle-cert\") pod \"isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:46.774266 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.774261    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gqn6\" (UniqueName: \"kubernetes.io/projected/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kube-api-access-8gqn6\") pod \"isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:46.774393 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.774324    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-proxy-tls\") pod \"isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:46.874797 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.874771    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:46.874901 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.874801    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kserve-provision-location\") pod \"isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:46.874901 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.874843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-cabundle-cert\") pod \"isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:46.874901 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.874872    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gqn6\" (UniqueName: \"kubernetes.io/projected/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kube-api-access-8gqn6\") pod \"isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:46.875081 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.874908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-proxy-tls\") pod \"isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:46.875233 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.875208    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kserve-provision-location\") pod \"isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:46.875532 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.875515    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-cabundle-cert\") pod \"isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:46.875579 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.875521    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:46.877197 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.877161    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-proxy-tls\") pod \"isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:46.883149 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.883130    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gqn6\" (UniqueName: \"kubernetes.io/projected/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kube-api-access-8gqn6\") pod \"isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:46.892194 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:46.892175    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:57:47.018507 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:47.018450    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:47.290887 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:47.290863    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"]
Sep 17 16:57:47.293606 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:57:47.293579    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda775d7e7_2d48_4957_80ac_5b57f1aa4d57.slice/crio-0399953e51b768bf0b29bb3e2bce6566acb17bd9643e3c94af9f239b744a359d WatchSource:0}: Error finding container 0399953e51b768bf0b29bb3e2bce6566acb17bd9643e3c94af9f239b744a359d: Status 404 returned error can't find the container with id 0399953e51b768bf0b29bb3e2bce6566acb17bd9643e3c94af9f239b744a359d
Sep 17 16:57:47.818787 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:47.818744    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" event={"ID":"a775d7e7-2d48-4957-80ac-5b57f1aa4d57","Type":"ContainerStarted","Data":"2ff934bbf94992089c57df772edc4779f2c4c823e3a95dc3c0714d723ec2564c"}
Sep 17 16:57:47.819296 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:47.818828    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" event={"ID":"a775d7e7-2d48-4957-80ac-5b57f1aa4d57","Type":"ContainerStarted","Data":"0399953e51b768bf0b29bb3e2bce6566acb17bd9643e3c94af9f239b744a359d"}
Sep 17 16:57:50.636407 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:50.636371    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 16:57:51.835928 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:51.835882    2565 generic.go:356] "Generic (PLEG): container finished" podID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerID="2ff934bbf94992089c57df772edc4779f2c4c823e3a95dc3c0714d723ec2564c" exitCode=0
Sep 17 16:57:51.836289 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:51.835941    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" event={"ID":"a775d7e7-2d48-4957-80ac-5b57f1aa4d57","Type":"ContainerDied","Data":"2ff934bbf94992089c57df772edc4779f2c4c823e3a95dc3c0714d723ec2564c"}
Sep 17 16:57:52.842697 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:52.842663    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" event={"ID":"a775d7e7-2d48-4957-80ac-5b57f1aa4d57","Type":"ContainerStarted","Data":"d0f9b00c8411b5446b9571d79ca118b19d1e055e019ae79707036650e99f3bf2"}
Sep 17 16:57:52.843107 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:52.842727    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" event={"ID":"a775d7e7-2d48-4957-80ac-5b57f1aa4d57","Type":"ContainerStarted","Data":"1ad5611a49256de704dc917edfeb6e5d7ef9853f54736c6c54fc34c76be66536"}
Sep 17 16:57:52.843107 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:52.842754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" event={"ID":"a775d7e7-2d48-4957-80ac-5b57f1aa4d57","Type":"ContainerStarted","Data":"9a6616b6ee6ee9f81ca6b7d079d0c3680768dcdddcefd6ea6898c60948fa1cb8"}
Sep 17 16:57:52.843107 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:52.843058    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:52.843282 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:52.843123    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:52.844286 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:52.844258    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:57:52.865086 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:52.865043    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podStartSLOduration=3.211204532 podStartE2EDuration="6.865028953s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.653824421s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:57:52.863607986 +0000 UTC m=+938.429719710" watchObservedRunningTime="2026-09-17 16:57:52.865028953 +0000 UTC m=+938.431140674"
Sep 17 16:57:53.847649 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:53.847608    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:53.848167 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:53.847766    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:57:53.849194 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:53.849162    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:57:54.851973 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:54.851933    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:57:54.852462 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:54.852429    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:57:59.857253 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:59.857226    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:57:59.857770 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:59.857739    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:57:59.857972 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:57:59.857950    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:58:00.636611 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:00.636571    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 16:58:09.858399 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:09.858356    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:58:09.858830 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:09.858805    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:58:10.636759 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:10.636719    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 16:58:19.857799 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:19.857760    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:58:19.858314 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:19.858290    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:58:20.641952 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:20.636946    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:58:29.857767 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:29.857674    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:58:29.858229 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:29.858111    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:58:39.857668 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:39.857625    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:58:39.858102 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:39.858066    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:58:41.593014 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:41.592986    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:58:42.821141 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:42.821106    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:58:48.159194 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:48.159167    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:58:49.857694 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:49.857655    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:58:49.858138 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:49.858113    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:58:59.679647 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:59.679613    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"]
Sep 17 16:58:59.680422 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:59.679906    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kserve-container" containerID="cri-o://a81de15aaee276aa33151beba04b84816add58366b3cf77f3d4f5a17d564c005" gracePeriod=30
Sep 17 16:58:59.680422 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:59.679971    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kube-rbac-proxy" containerID="cri-o://2d60c713ba68ed154dcc7d00043ba0e0cfb60c8be63cc93a4f5f444aebbd1c4e" gracePeriod=30
Sep 17 16:58:59.689281 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:59.689254    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:58:59.744225 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:59.744205    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:58:59.858044 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:59.858009    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:58:59.858463 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:58:59.858439    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:59:00.110408 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:00.110344    2565 generic.go:356] "Generic (PLEG): container finished" podID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerID="2d60c713ba68ed154dcc7d00043ba0e0cfb60c8be63cc93a4f5f444aebbd1c4e" exitCode=2
Sep 17 16:59:00.110408 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:00.110401    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" event={"ID":"4d11abf8-027d-4f87-8a06-e57af2fde6bf","Type":"ContainerDied","Data":"2d60c713ba68ed154dcc7d00043ba0e0cfb60c8be63cc93a4f5f444aebbd1c4e"}
Sep 17 16:59:00.631453 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:00.631415    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 17 16:59:00.636019 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:00.635988    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 17 16:59:03.422260 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:03.422237    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:59:03.531235 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:03.531168    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d11abf8-027d-4f87-8a06-e57af2fde6bf-cabundle-cert\") pod \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") "
Sep 17 16:59:03.531235 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:03.531210    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wbzh\" (UniqueName: \"kubernetes.io/projected/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kube-api-access-2wbzh\") pod \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") "
Sep 17 16:59:03.531235 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:03.531236    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d11abf8-027d-4f87-8a06-e57af2fde6bf-proxy-tls\") pod \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") "
Sep 17 16:59:03.531485 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:03.531260    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kserve-provision-location\") pod \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") "
Sep 17 16:59:03.531485 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:03.531310    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kube-rbac-proxy-sar-config\") pod \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\" (UID: \"4d11abf8-027d-4f87-8a06-e57af2fde6bf\") "
Sep 17 16:59:03.531635 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:03.531608    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d11abf8-027d-4f87-8a06-e57af2fde6bf" (UID: "4d11abf8-027d-4f87-8a06-e57af2fde6bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:59:03.531694 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:03.531675    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d11abf8-027d-4f87-8a06-e57af2fde6bf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4d11abf8-027d-4f87-8a06-e57af2fde6bf" (UID: "4d11abf8-027d-4f87-8a06-e57af2fde6bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:59:03.531745 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:03.531723    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4d11abf8-027d-4f87-8a06-e57af2fde6bf" (UID: "4d11abf8-027d-4f87-8a06-e57af2fde6bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:59:03.533244 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:03.533222    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d11abf8-027d-4f87-8a06-e57af2fde6bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d11abf8-027d-4f87-8a06-e57af2fde6bf" (UID: "4d11abf8-027d-4f87-8a06-e57af2fde6bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:59:03.533342 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:03.533271    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kube-api-access-2wbzh" (OuterVolumeSpecName: "kube-api-access-2wbzh") pod "4d11abf8-027d-4f87-8a06-e57af2fde6bf" (UID: "4d11abf8-027d-4f87-8a06-e57af2fde6bf"). InnerVolumeSpecName "kube-api-access-2wbzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:59:03.632257 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:03.632233    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d11abf8-027d-4f87-8a06-e57af2fde6bf-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:59:03.632257 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:03.632252    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wbzh\" (UniqueName: \"kubernetes.io/projected/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kube-api-access-2wbzh\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:59:03.632391 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:03.632264    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d11abf8-027d-4f87-8a06-e57af2fde6bf-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:59:03.632391 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:03.632274    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:59:03.632391 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:03.632282    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d11abf8-027d-4f87-8a06-e57af2fde6bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:59:04.129040 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.129003    2565 generic.go:356] "Generic (PLEG): container finished" podID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerID="a81de15aaee276aa33151beba04b84816add58366b3cf77f3d4f5a17d564c005" exitCode=0
Sep 17 16:59:04.129210 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.129061    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"
Sep 17 16:59:04.129210 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.129088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" event={"ID":"4d11abf8-027d-4f87-8a06-e57af2fde6bf","Type":"ContainerDied","Data":"a81de15aaee276aa33151beba04b84816add58366b3cf77f3d4f5a17d564c005"}
Sep 17 16:59:04.129210 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.129150    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt" event={"ID":"4d11abf8-027d-4f87-8a06-e57af2fde6bf","Type":"ContainerDied","Data":"1d7e39f737f50e716abefeb6247538c2da2ff43317576c6ef28f02144319f7c1"}
Sep 17 16:59:04.129210 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.129177    2565 scope.go:117] "RemoveContainer" containerID="2d60c713ba68ed154dcc7d00043ba0e0cfb60c8be63cc93a4f5f444aebbd1c4e"
Sep 17 16:59:04.144033 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.144014    2565 scope.go:117] "RemoveContainer" containerID="a81de15aaee276aa33151beba04b84816add58366b3cf77f3d4f5a17d564c005"
Sep 17 16:59:04.151307 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.151290    2565 scope.go:117] "RemoveContainer" containerID="d3f036ccbc71a3b6e7213488f6376f929bcea30a98455d3b7338dbdb629f194d"
Sep 17 16:59:04.153628 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.153607    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"]
Sep 17 16:59:04.158051 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.158030    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-42374c-predictor-798b4f4b89-bb7vt"]
Sep 17 16:59:04.158744 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.158725    2565 scope.go:117] "RemoveContainer" containerID="2d60c713ba68ed154dcc7d00043ba0e0cfb60c8be63cc93a4f5f444aebbd1c4e"
Sep 17 16:59:04.159092 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:59:04.159074    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d60c713ba68ed154dcc7d00043ba0e0cfb60c8be63cc93a4f5f444aebbd1c4e\": container with ID starting with 2d60c713ba68ed154dcc7d00043ba0e0cfb60c8be63cc93a4f5f444aebbd1c4e not found: ID does not exist" containerID="2d60c713ba68ed154dcc7d00043ba0e0cfb60c8be63cc93a4f5f444aebbd1c4e"
Sep 17 16:59:04.159171 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.159096    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d60c713ba68ed154dcc7d00043ba0e0cfb60c8be63cc93a4f5f444aebbd1c4e"} err="failed to get container status \"2d60c713ba68ed154dcc7d00043ba0e0cfb60c8be63cc93a4f5f444aebbd1c4e\": rpc error: code = NotFound desc = could not find container \"2d60c713ba68ed154dcc7d00043ba0e0cfb60c8be63cc93a4f5f444aebbd1c4e\": container with ID starting with 2d60c713ba68ed154dcc7d00043ba0e0cfb60c8be63cc93a4f5f444aebbd1c4e not found: ID does not exist"
Sep 17 16:59:04.159171 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.159112    2565 scope.go:117] "RemoveContainer" containerID="a81de15aaee276aa33151beba04b84816add58366b3cf77f3d4f5a17d564c005"
Sep 17 16:59:04.159356 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:59:04.159341    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a81de15aaee276aa33151beba04b84816add58366b3cf77f3d4f5a17d564c005\": container with ID starting with a81de15aaee276aa33151beba04b84816add58366b3cf77f3d4f5a17d564c005 not found: ID does not exist" containerID="a81de15aaee276aa33151beba04b84816add58366b3cf77f3d4f5a17d564c005"
Sep 17 16:59:04.159397 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.159360    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a81de15aaee276aa33151beba04b84816add58366b3cf77f3d4f5a17d564c005"} err="failed to get container status \"a81de15aaee276aa33151beba04b84816add58366b3cf77f3d4f5a17d564c005\": rpc error: code = NotFound desc = could not find container \"a81de15aaee276aa33151beba04b84816add58366b3cf77f3d4f5a17d564c005\": container with ID starting with a81de15aaee276aa33151beba04b84816add58366b3cf77f3d4f5a17d564c005 not found: ID does not exist"
Sep 17 16:59:04.159397 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.159372    2565 scope.go:117] "RemoveContainer" containerID="d3f036ccbc71a3b6e7213488f6376f929bcea30a98455d3b7338dbdb629f194d"
Sep 17 16:59:04.159624 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:59:04.159604    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f036ccbc71a3b6e7213488f6376f929bcea30a98455d3b7338dbdb629f194d\": container with ID starting with d3f036ccbc71a3b6e7213488f6376f929bcea30a98455d3b7338dbdb629f194d not found: ID does not exist" containerID="d3f036ccbc71a3b6e7213488f6376f929bcea30a98455d3b7338dbdb629f194d"
Sep 17 16:59:04.159674 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.159630    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f036ccbc71a3b6e7213488f6376f929bcea30a98455d3b7338dbdb629f194d"} err="failed to get container status \"d3f036ccbc71a3b6e7213488f6376f929bcea30a98455d3b7338dbdb629f194d\": rpc error: code = NotFound desc = could not find container \"d3f036ccbc71a3b6e7213488f6376f929bcea30a98455d3b7338dbdb629f194d\": container with ID starting with d3f036ccbc71a3b6e7213488f6376f929bcea30a98455d3b7338dbdb629f194d not found: ID does not exist"
Sep 17 16:59:04.824561 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.824527    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" path="/var/lib/kubelet/pods/4d11abf8-027d-4f87-8a06-e57af2fde6bf/volumes"
Sep 17 16:59:04.864633 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:04.864609    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:59:05.042889 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:05.042863    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:59:09.858295 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:09.858270    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:59:09.858741 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:09.858492    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:59:14.892131 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:14.892102    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:59:20.054372 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:20.054340    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:59:20.243632 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:20.243603    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:59:21.690361 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:21.690324    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn_550bae87-0b56-4165-a460-ff9e4ce671dc/kserve-container/0.log"
Sep 17 16:59:21.816987 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:21.816787    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"]
Sep 17 16:59:21.817159 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:21.817116    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container" containerID="cri-o://9a6616b6ee6ee9f81ca6b7d079d0c3680768dcdddcefd6ea6898c60948fa1cb8" gracePeriod=30
Sep 17 16:59:21.817481 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:21.817360    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="agent" containerID="cri-o://d0f9b00c8411b5446b9571d79ca118b19d1e055e019ae79707036650e99f3bf2" gracePeriod=30
Sep 17 16:59:21.818078 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:21.817677    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kube-rbac-proxy" containerID="cri-o://1ad5611a49256de704dc917edfeb6e5d7ef9853f54736c6c54fc34c76be66536" gracePeriod=30
Sep 17 16:59:21.884807 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:21.884779    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:59:21.899964 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:21.899936    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"]
Sep 17 16:59:21.900171 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:21.900148    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn" podUID="550bae87-0b56-4165-a460-ff9e4ce671dc" containerName="kserve-container" containerID="cri-o://45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432" gracePeriod=30
Sep 17 16:59:21.900219 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:21.900181    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn" podUID="550bae87-0b56-4165-a460-ff9e4ce671dc" containerName="kube-rbac-proxy" containerID="cri-o://07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de" gracePeriod=30
Sep 17 16:59:21.974647 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:21.974585    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:59:22.024468 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.024443    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"]
Sep 17 16:59:22.024769 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.024758    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kserve-container"
Sep 17 16:59:22.024814 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.024771    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kserve-container"
Sep 17 16:59:22.024814 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.024787    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="storage-initializer"
Sep 17 16:59:22.024814 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.024793    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="storage-initializer"
Sep 17 16:59:22.024814 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.024799    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kube-rbac-proxy"
Sep 17 16:59:22.024814 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.024804    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kube-rbac-proxy"
Sep 17 16:59:22.024989 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.024851    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kube-rbac-proxy"
Sep 17 16:59:22.024989 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.024861    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d11abf8-027d-4f87-8a06-e57af2fde6bf" containerName="kserve-container"
Sep 17 16:59:22.029284 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.029259    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.032226 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.032198    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-f447b-predictor-serving-cert\""
Sep 17 16:59:22.032460 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.032442    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-f447b-kube-rbac-proxy-sar-config\""
Sep 17 16:59:22.037756 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.037680    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"]
Sep 17 16:59:22.042602 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.042580    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:59:22.151981 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.151953    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:59:22.166641 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.166619    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6822971d-be41-4680-8fe1-ff883636d593-proxy-tls\") pod \"isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.166749 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.166656    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw67n\" (UniqueName: \"kubernetes.io/projected/6822971d-be41-4680-8fe1-ff883636d593-kube-api-access-zw67n\") pod \"isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.166749 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.166689    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6822971d-be41-4680-8fe1-ff883636d593-cabundle-cert\") pod \"isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.166817 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.166776    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6822971d-be41-4680-8fe1-ff883636d593-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.166817 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.166806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6822971d-be41-4680-8fe1-ff883636d593-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.203026 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.203004    2565 generic.go:356] "Generic (PLEG): container finished" podID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerID="1ad5611a49256de704dc917edfeb6e5d7ef9853f54736c6c54fc34c76be66536" exitCode=2
Sep 17 16:59:22.203113 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.203081    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" event={"ID":"a775d7e7-2d48-4957-80ac-5b57f1aa4d57","Type":"ContainerDied","Data":"1ad5611a49256de704dc917edfeb6e5d7ef9853f54736c6c54fc34c76be66536"}
Sep 17 16:59:22.204529 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.204503    2565 generic.go:356] "Generic (PLEG): container finished" podID="550bae87-0b56-4165-a460-ff9e4ce671dc" containerID="07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de" exitCode=2
Sep 17 16:59:22.204529 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.204529    2565 generic.go:356] "Generic (PLEG): container finished" podID="550bae87-0b56-4165-a460-ff9e4ce671dc" containerID="45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432" exitCode=2
Sep 17 16:59:22.204645 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.204554    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"
Sep 17 16:59:22.204645 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.204581    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn" event={"ID":"550bae87-0b56-4165-a460-ff9e4ce671dc","Type":"ContainerDied","Data":"07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de"}
Sep 17 16:59:22.204645 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.204626    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn" event={"ID":"550bae87-0b56-4165-a460-ff9e4ce671dc","Type":"ContainerDied","Data":"45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432"}
Sep 17 16:59:22.204747 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.204646    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn" event={"ID":"550bae87-0b56-4165-a460-ff9e4ce671dc","Type":"ContainerDied","Data":"9e8080a803d12a4280e013949b60bae82324f29d01c1d6f65947b9d00548105a"}
Sep 17 16:59:22.204747 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.204669    2565 scope.go:117] "RemoveContainer" containerID="07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de"
Sep 17 16:59:22.213968 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.213952    2565 scope.go:117] "RemoveContainer" containerID="45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432"
Sep 17 16:59:22.220861 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.220842    2565 scope.go:117] "RemoveContainer" containerID="07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de"
Sep 17 16:59:22.221146 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:59:22.221127    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de\": container with ID starting with 07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de not found: ID does not exist" containerID="07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de"
Sep 17 16:59:22.221207 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.221154    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de"} err="failed to get container status \"07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de\": rpc error: code = NotFound desc = could not find container \"07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de\": container with ID starting with 07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de not found: ID does not exist"
Sep 17 16:59:22.221207 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.221170    2565 scope.go:117] "RemoveContainer" containerID="45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432"
Sep 17 16:59:22.221403 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:59:22.221386    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432\": container with ID starting with 45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432 not found: ID does not exist" containerID="45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432"
Sep 17 16:59:22.221468 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.221412    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432"} err="failed to get container status \"45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432\": rpc error: code = NotFound desc = could not find container \"45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432\": container with ID starting with 45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432 not found: ID does not exist"
Sep 17 16:59:22.221468 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.221434    2565 scope.go:117] "RemoveContainer" containerID="07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de"
Sep 17 16:59:22.221700 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.221682    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de"} err="failed to get container status \"07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de\": rpc error: code = NotFound desc = could not find container \"07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de\": container with ID starting with 07f965e9a0ebc6dfd04477f71b27f1e1109ebd4834d714e095ad5e542274e2de not found: ID does not exist"
Sep 17 16:59:22.221757 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.221703    2565 scope.go:117] "RemoveContainer" containerID="45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432"
Sep 17 16:59:22.221937 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.221902    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432"} err="failed to get container status \"45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432\": rpc error: code = NotFound desc = could not find container \"45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432\": container with ID starting with 45e9e9c969a91020a11e89e6dd063fd6a78d0d2696f6fe9d644051ae8e76f432 not found: ID does not exist"
Sep 17 16:59:22.267772 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.267716    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/550bae87-0b56-4165-a460-ff9e4ce671dc-proxy-tls\") pod \"550bae87-0b56-4165-a460-ff9e4ce671dc\" (UID: \"550bae87-0b56-4165-a460-ff9e4ce671dc\") "
Sep 17 16:59:22.267772 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.267761    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmdst\" (UniqueName: \"kubernetes.io/projected/550bae87-0b56-4165-a460-ff9e4ce671dc-kube-api-access-zmdst\") pod \"550bae87-0b56-4165-a460-ff9e4ce671dc\" (UID: \"550bae87-0b56-4165-a460-ff9e4ce671dc\") "
Sep 17 16:59:22.267941 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.267836    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/550bae87-0b56-4165-a460-ff9e4ce671dc-kube-rbac-proxy-sar-config\") pod \"550bae87-0b56-4165-a460-ff9e4ce671dc\" (UID: \"550bae87-0b56-4165-a460-ff9e4ce671dc\") "
Sep 17 16:59:22.268016 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.267993    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6822971d-be41-4680-8fe1-ff883636d593-cabundle-cert\") pod \"isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.268089 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.268071    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6822971d-be41-4680-8fe1-ff883636d593-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.268149 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.268114    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6822971d-be41-4680-8fe1-ff883636d593-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.268203 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.268173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6822971d-be41-4680-8fe1-ff883636d593-proxy-tls\") pod \"isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.268260 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.268206    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550bae87-0b56-4165-a460-ff9e4ce671dc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "550bae87-0b56-4165-a460-ff9e4ce671dc" (UID: "550bae87-0b56-4165-a460-ff9e4ce671dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:59:22.268260 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.268215    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw67n\" (UniqueName: \"kubernetes.io/projected/6822971d-be41-4680-8fe1-ff883636d593-kube-api-access-zw67n\") pod \"isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.268359 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.268290    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/550bae87-0b56-4165-a460-ff9e4ce671dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:59:22.268414 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:59:22.268380    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-serving-cert: secret "isvc-sklearn-scale-raw-f447b-predictor-serving-cert" not found
Sep 17 16:59:22.268466 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:59:22.268443    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6822971d-be41-4680-8fe1-ff883636d593-proxy-tls podName:6822971d-be41-4680-8fe1-ff883636d593 nodeName:}" failed. No retries permitted until 2026-09-17 16:59:22.768425703 +0000 UTC m=+1028.334537407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6822971d-be41-4680-8fe1-ff883636d593-proxy-tls") pod "isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" (UID: "6822971d-be41-4680-8fe1-ff883636d593") : secret "isvc-sklearn-scale-raw-f447b-predictor-serving-cert" not found
Sep 17 16:59:22.268528 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.268473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6822971d-be41-4680-8fe1-ff883636d593-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.268744 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.268724    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6822971d-be41-4680-8fe1-ff883636d593-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.268744 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.268738    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6822971d-be41-4680-8fe1-ff883636d593-cabundle-cert\") pod \"isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.270239 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.270219    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/550bae87-0b56-4165-a460-ff9e4ce671dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "550bae87-0b56-4165-a460-ff9e4ce671dc" (UID: "550bae87-0b56-4165-a460-ff9e4ce671dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:59:22.270295 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.270240    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550bae87-0b56-4165-a460-ff9e4ce671dc-kube-api-access-zmdst" (OuterVolumeSpecName: "kube-api-access-zmdst") pod "550bae87-0b56-4165-a460-ff9e4ce671dc" (UID: "550bae87-0b56-4165-a460-ff9e4ce671dc"). InnerVolumeSpecName "kube-api-access-zmdst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:59:22.278631 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.278615    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw67n\" (UniqueName: \"kubernetes.io/projected/6822971d-be41-4680-8fe1-ff883636d593-kube-api-access-zw67n\") pod \"isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.369098 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.369077    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/550bae87-0b56-4165-a460-ff9e4ce671dc-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:59:22.369098 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.369097    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmdst\" (UniqueName: \"kubernetes.io/projected/550bae87-0b56-4165-a460-ff9e4ce671dc-kube-api-access-zmdst\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:59:22.486446 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.486426    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:59:22.525112 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.525063    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"]
Sep 17 16:59:22.528495 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.528475    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-c61ee-predictor-54fdf9d498-kr7nn"]
Sep 17 16:59:22.772091 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.772062    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6822971d-be41-4680-8fe1-ff883636d593-proxy-tls\") pod \"isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.774221 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.774202    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6822971d-be41-4680-8fe1-ff883636d593-proxy-tls\") pod \"isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:22.823006 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.822952    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="550bae87-0b56-4165-a460-ff9e4ce671dc" path="/var/lib/kubelet/pods/550bae87-0b56-4165-a460-ff9e4ce671dc/volumes"
Sep 17 16:59:22.941760 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:22.941738    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:23.216296 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:23.216275    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"]
Sep 17 16:59:23.218609 ip-10-0-132-85 kubenswrapper[2565]: W0917 16:59:23.218583    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6822971d_be41_4680_8fe1_ff883636d593.slice/crio-55f121b97c4f8490eb6766747131bc1f358f6b44e90aa1843d2f3b107b02f834 WatchSource:0}: Error finding container 55f121b97c4f8490eb6766747131bc1f358f6b44e90aa1843d2f3b107b02f834: Status 404 returned error can't find the container with id 55f121b97c4f8490eb6766747131bc1f358f6b44e90aa1843d2f3b107b02f834
Sep 17 16:59:24.215019 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:24.214985    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" event={"ID":"6822971d-be41-4680-8fe1-ff883636d593","Type":"ContainerStarted","Data":"23151ecbf6f2b1704c5c125ee2bc31f29c334ff0e708153d23fccc5efb8e23b9"}
Sep 17 16:59:24.215434 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:24.215049    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" event={"ID":"6822971d-be41-4680-8fe1-ff883636d593","Type":"ContainerStarted","Data":"55f121b97c4f8490eb6766747131bc1f358f6b44e90aa1843d2f3b107b02f834"}
Sep 17 16:59:24.852030 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:24.851984    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 17 16:59:26.225805 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:26.225777    2565 generic.go:356] "Generic (PLEG): container finished" podID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerID="9a6616b6ee6ee9f81ca6b7d079d0c3680768dcdddcefd6ea6898c60948fa1cb8" exitCode=0
Sep 17 16:59:26.226149 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:26.225843    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" event={"ID":"a775d7e7-2d48-4957-80ac-5b57f1aa4d57","Type":"ContainerDied","Data":"9a6616b6ee6ee9f81ca6b7d079d0c3680768dcdddcefd6ea6898c60948fa1cb8"}
Sep 17 16:59:27.232201 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:27.232168    2565 generic.go:356] "Generic (PLEG): container finished" podID="6822971d-be41-4680-8fe1-ff883636d593" containerID="23151ecbf6f2b1704c5c125ee2bc31f29c334ff0e708153d23fccc5efb8e23b9" exitCode=0
Sep 17 16:59:27.232507 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:27.232244    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" event={"ID":"6822971d-be41-4680-8fe1-ff883636d593","Type":"ContainerDied","Data":"23151ecbf6f2b1704c5c125ee2bc31f29c334ff0e708153d23fccc5efb8e23b9"}
Sep 17 16:59:28.239379 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:28.239350    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" event={"ID":"6822971d-be41-4680-8fe1-ff883636d593","Type":"ContainerStarted","Data":"414aa749f586f53fd0aa813d808701076dff925be6ae2290c60a12b586efd900"}
Sep 17 16:59:28.239777 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:28.239409    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" event={"ID":"6822971d-be41-4680-8fe1-ff883636d593","Type":"ContainerStarted","Data":"ff29d3f189d5a30a213c2e22cfd33b28a071796270380faa55c37f35370c3598"}
Sep 17 16:59:28.239777 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:28.239634    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:28.239777 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:28.239774    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:28.241185 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:28.241155    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:59:28.260045 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:28.259997    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podStartSLOduration=2.378086432 podStartE2EDuration="6.259984182s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.88189775s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:59:28.257474036 +0000 UTC m=+1033.823585759" watchObservedRunningTime="2026-09-17 16:59:28.259984182 +0000 UTC m=+1033.826095904"
Sep 17 16:59:29.243091 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:29.243043    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:59:29.852078 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:29.852043    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 17 16:59:29.857928 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:29.857873    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:59:29.858209 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:29.858185    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:59:33.680984 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:33.680949    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 16:59:34.248163 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:34.248132    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 16:59:34.248751 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:34.248713    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:59:34.852326 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:34.852291    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 17 16:59:34.852687 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:34.852396    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:59:39.852452 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:39.852413    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 17 16:59:39.857793 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:39.857754    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:59:39.858087 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:39.858059    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:59:44.248577 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:44.248538    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:59:44.852488 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:44.852446    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 17 16:59:49.852554 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:49.852513    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 17 16:59:49.857962 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:49.857930    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 16:59:49.858111 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:49.858069    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:59:49.858329 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:49.858311    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:59:49.858426 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:49.858412    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:59:51.991408 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:51.991385    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:59:52.077143 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.077073    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-proxy-tls\") pod \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") "
Sep 17 16:59:52.077143 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.077108    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kserve-provision-location\") pod \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") "
Sep 17 16:59:52.077334 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.077163    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kube-rbac-proxy-sar-config\") pod \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") "
Sep 17 16:59:52.077334 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.077186    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gqn6\" (UniqueName: \"kubernetes.io/projected/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kube-api-access-8gqn6\") pod \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") "
Sep 17 16:59:52.077334 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.077260    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-cabundle-cert\") pod \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\" (UID: \"a775d7e7-2d48-4957-80ac-5b57f1aa4d57\") "
Sep 17 16:59:52.077502 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.077440    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a775d7e7-2d48-4957-80ac-5b57f1aa4d57" (UID: "a775d7e7-2d48-4957-80ac-5b57f1aa4d57"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:59:52.077624 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.077592    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a775d7e7-2d48-4957-80ac-5b57f1aa4d57" (UID: "a775d7e7-2d48-4957-80ac-5b57f1aa4d57"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:59:52.077741 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.077713    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a775d7e7-2d48-4957-80ac-5b57f1aa4d57" (UID: "a775d7e7-2d48-4957-80ac-5b57f1aa4d57"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:59:52.079215 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.079186    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kube-api-access-8gqn6" (OuterVolumeSpecName: "kube-api-access-8gqn6") pod "a775d7e7-2d48-4957-80ac-5b57f1aa4d57" (UID: "a775d7e7-2d48-4957-80ac-5b57f1aa4d57"). InnerVolumeSpecName "kube-api-access-8gqn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:59:52.079300 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.079237    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a775d7e7-2d48-4957-80ac-5b57f1aa4d57" (UID: "a775d7e7-2d48-4957-80ac-5b57f1aa4d57"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:59:52.177868 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.177847    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:59:52.177868 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.177867    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:59:52.178024 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.177877    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:59:52.178024 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.177885    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gqn6\" (UniqueName: \"kubernetes.io/projected/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-kube-api-access-8gqn6\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:59:52.178024 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.177893    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a775d7e7-2d48-4957-80ac-5b57f1aa4d57-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 16:59:52.335116 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.335055    2565 generic.go:356] "Generic (PLEG): container finished" podID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerID="d0f9b00c8411b5446b9571d79ca118b19d1e055e019ae79707036650e99f3bf2" exitCode=0
Sep 17 16:59:52.335116 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.335094    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" event={"ID":"a775d7e7-2d48-4957-80ac-5b57f1aa4d57","Type":"ContainerDied","Data":"d0f9b00c8411b5446b9571d79ca118b19d1e055e019ae79707036650e99f3bf2"}
Sep 17 16:59:52.335240 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.335137    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"
Sep 17 16:59:52.335240 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.335182    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm" event={"ID":"a775d7e7-2d48-4957-80ac-5b57f1aa4d57","Type":"ContainerDied","Data":"0399953e51b768bf0b29bb3e2bce6566acb17bd9643e3c94af9f239b744a359d"}
Sep 17 16:59:52.335240 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.335220    2565 scope.go:117] "RemoveContainer" containerID="d0f9b00c8411b5446b9571d79ca118b19d1e055e019ae79707036650e99f3bf2"
Sep 17 16:59:52.343517 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.343498    2565 scope.go:117] "RemoveContainer" containerID="1ad5611a49256de704dc917edfeb6e5d7ef9853f54736c6c54fc34c76be66536"
Sep 17 16:59:52.351360 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.351344    2565 scope.go:117] "RemoveContainer" containerID="9a6616b6ee6ee9f81ca6b7d079d0c3680768dcdddcefd6ea6898c60948fa1cb8"
Sep 17 16:59:52.357714 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.357693    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"]
Sep 17 16:59:52.359089 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.359061    2565 scope.go:117] "RemoveContainer" containerID="2ff934bbf94992089c57df772edc4779f2c4c823e3a95dc3c0714d723ec2564c"
Sep 17 16:59:52.362183 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.362160    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-c61ee-predictor-6d9cdc4d99-kfzjm"]
Sep 17 16:59:52.366567 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.366550    2565 scope.go:117] "RemoveContainer" containerID="d0f9b00c8411b5446b9571d79ca118b19d1e055e019ae79707036650e99f3bf2"
Sep 17 16:59:52.366846 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:59:52.366827    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f9b00c8411b5446b9571d79ca118b19d1e055e019ae79707036650e99f3bf2\": container with ID starting with d0f9b00c8411b5446b9571d79ca118b19d1e055e019ae79707036650e99f3bf2 not found: ID does not exist" containerID="d0f9b00c8411b5446b9571d79ca118b19d1e055e019ae79707036650e99f3bf2"
Sep 17 16:59:52.366896 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.366853    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f9b00c8411b5446b9571d79ca118b19d1e055e019ae79707036650e99f3bf2"} err="failed to get container status \"d0f9b00c8411b5446b9571d79ca118b19d1e055e019ae79707036650e99f3bf2\": rpc error: code = NotFound desc = could not find container \"d0f9b00c8411b5446b9571d79ca118b19d1e055e019ae79707036650e99f3bf2\": container with ID starting with d0f9b00c8411b5446b9571d79ca118b19d1e055e019ae79707036650e99f3bf2 not found: ID does not exist"
Sep 17 16:59:52.366896 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.366871    2565 scope.go:117] "RemoveContainer" containerID="1ad5611a49256de704dc917edfeb6e5d7ef9853f54736c6c54fc34c76be66536"
Sep 17 16:59:52.367132 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:59:52.367117    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad5611a49256de704dc917edfeb6e5d7ef9853f54736c6c54fc34c76be66536\": container with ID starting with 1ad5611a49256de704dc917edfeb6e5d7ef9853f54736c6c54fc34c76be66536 not found: ID does not exist" containerID="1ad5611a49256de704dc917edfeb6e5d7ef9853f54736c6c54fc34c76be66536"
Sep 17 16:59:52.367171 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.367137    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad5611a49256de704dc917edfeb6e5d7ef9853f54736c6c54fc34c76be66536"} err="failed to get container status \"1ad5611a49256de704dc917edfeb6e5d7ef9853f54736c6c54fc34c76be66536\": rpc error: code = NotFound desc = could not find container \"1ad5611a49256de704dc917edfeb6e5d7ef9853f54736c6c54fc34c76be66536\": container with ID starting with 1ad5611a49256de704dc917edfeb6e5d7ef9853f54736c6c54fc34c76be66536 not found: ID does not exist"
Sep 17 16:59:52.367171 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.367153    2565 scope.go:117] "RemoveContainer" containerID="9a6616b6ee6ee9f81ca6b7d079d0c3680768dcdddcefd6ea6898c60948fa1cb8"
Sep 17 16:59:52.367398 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:59:52.367377    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a6616b6ee6ee9f81ca6b7d079d0c3680768dcdddcefd6ea6898c60948fa1cb8\": container with ID starting with 9a6616b6ee6ee9f81ca6b7d079d0c3680768dcdddcefd6ea6898c60948fa1cb8 not found: ID does not exist" containerID="9a6616b6ee6ee9f81ca6b7d079d0c3680768dcdddcefd6ea6898c60948fa1cb8"
Sep 17 16:59:52.367442 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.367403    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6616b6ee6ee9f81ca6b7d079d0c3680768dcdddcefd6ea6898c60948fa1cb8"} err="failed to get container status \"9a6616b6ee6ee9f81ca6b7d079d0c3680768dcdddcefd6ea6898c60948fa1cb8\": rpc error: code = NotFound desc = could not find container \"9a6616b6ee6ee9f81ca6b7d079d0c3680768dcdddcefd6ea6898c60948fa1cb8\": container with ID starting with 9a6616b6ee6ee9f81ca6b7d079d0c3680768dcdddcefd6ea6898c60948fa1cb8 not found: ID does not exist"
Sep 17 16:59:52.367442 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.367418    2565 scope.go:117] "RemoveContainer" containerID="2ff934bbf94992089c57df772edc4779f2c4c823e3a95dc3c0714d723ec2564c"
Sep 17 16:59:52.367667 ip-10-0-132-85 kubenswrapper[2565]: E0917 16:59:52.367648    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ff934bbf94992089c57df772edc4779f2c4c823e3a95dc3c0714d723ec2564c\": container with ID starting with 2ff934bbf94992089c57df772edc4779f2c4c823e3a95dc3c0714d723ec2564c not found: ID does not exist" containerID="2ff934bbf94992089c57df772edc4779f2c4c823e3a95dc3c0714d723ec2564c"
Sep 17 16:59:52.367720 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.367674    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ff934bbf94992089c57df772edc4779f2c4c823e3a95dc3c0714d723ec2564c"} err="failed to get container status \"2ff934bbf94992089c57df772edc4779f2c4c823e3a95dc3c0714d723ec2564c\": rpc error: code = NotFound desc = could not find container \"2ff934bbf94992089c57df772edc4779f2c4c823e3a95dc3c0714d723ec2564c\": container with ID starting with 2ff934bbf94992089c57df772edc4779f2c4c823e3a95dc3c0714d723ec2564c not found: ID does not exist"
Sep 17 16:59:52.823717 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:52.823686    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" path="/var/lib/kubelet/pods/a775d7e7-2d48-4957-80ac-5b57f1aa4d57/volumes"
Sep 17 16:59:54.249364 ip-10-0-132-85 kubenswrapper[2565]: I0917 16:59:54.249323    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:00:03.601328 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:03.601248    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:00:04.249326 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:04.249289    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:00:14.249185 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:14.249149    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:00:24.248809 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:24.248765    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:00:30.412030 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:30.411990    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:00:34.249200 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:34.249163    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:00:44.249166 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:44.249127    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:00:50.187106 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:50.187077    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:00:54.249276 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:54.249228    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:00:55.311173 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.311140    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:00:55.369529 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.369504    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"]
Sep 17 17:00:55.369845 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.369824    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="agent"
Sep 17 17:00:55.369845 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.369845    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="agent"
Sep 17 17:00:55.370045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.369863    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kube-rbac-proxy"
Sep 17 17:00:55.370045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.369869    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kube-rbac-proxy"
Sep 17 17:00:55.370045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.369881    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="550bae87-0b56-4165-a460-ff9e4ce671dc" containerName="kube-rbac-proxy"
Sep 17 17:00:55.370045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.369890    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="550bae87-0b56-4165-a460-ff9e4ce671dc" containerName="kube-rbac-proxy"
Sep 17 17:00:55.370045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.369898    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="storage-initializer"
Sep 17 17:00:55.370045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.369906    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="storage-initializer"
Sep 17 17:00:55.370045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.369930    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container"
Sep 17 17:00:55.370045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.369937    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container"
Sep 17 17:00:55.370045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.369947    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="550bae87-0b56-4165-a460-ff9e4ce671dc" containerName="kserve-container"
Sep 17 17:00:55.370045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.369952    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="550bae87-0b56-4165-a460-ff9e4ce671dc" containerName="kserve-container"
Sep 17 17:00:55.370045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.370020    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="550bae87-0b56-4165-a460-ff9e4ce671dc" containerName="kserve-container"
Sep 17 17:00:55.370045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.370028    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="550bae87-0b56-4165-a460-ff9e4ce671dc" containerName="kube-rbac-proxy"
Sep 17 17:00:55.370045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.370033    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kserve-container"
Sep 17 17:00:55.370045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.370040    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="kube-rbac-proxy"
Sep 17 17:00:55.370045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.370051    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a775d7e7-2d48-4957-80ac-5b57f1aa4d57" containerName="agent"
Sep 17 17:00:55.373239 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.373223    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:55.375703 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.375683    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-60bd0-predictor-serving-cert\""
Sep 17 17:00:55.375822 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.375743    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-60bd0-kube-rbac-proxy-sar-config\""
Sep 17 17:00:55.383351 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.383329    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"]
Sep 17 17:00:55.408157 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.408130    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/111e456f-dabb-4d46-9964-db0fe2dda6fe-proxy-tls\") pod \"raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:55.408254 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.408174    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdx5\" (UniqueName: \"kubernetes.io/projected/111e456f-dabb-4d46-9964-db0fe2dda6fe-kube-api-access-8zdx5\") pod \"raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:55.408254 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.408213    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/111e456f-dabb-4d46-9964-db0fe2dda6fe-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:55.408254 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.408230    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/111e456f-dabb-4d46-9964-db0fe2dda6fe-kserve-provision-location\") pod \"raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:55.408398 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.408263    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/111e456f-dabb-4d46-9964-db0fe2dda6fe-cabundle-cert\") pod \"raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:55.509389 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.509366    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/111e456f-dabb-4d46-9964-db0fe2dda6fe-cabundle-cert\") pod \"raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:55.509493 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.509399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/111e456f-dabb-4d46-9964-db0fe2dda6fe-proxy-tls\") pod \"raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:55.509493 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.509425    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdx5\" (UniqueName: \"kubernetes.io/projected/111e456f-dabb-4d46-9964-db0fe2dda6fe-kube-api-access-8zdx5\") pod \"raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:55.509493 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.509452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/111e456f-dabb-4d46-9964-db0fe2dda6fe-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:55.509493 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.509469    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/111e456f-dabb-4d46-9964-db0fe2dda6fe-kserve-provision-location\") pod \"raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:55.509718 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:00:55.509527    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-serving-cert: secret "raw-sklearn-runtime-60bd0-predictor-serving-cert" not found
Sep 17 17:00:55.509718 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:00:55.509595    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111e456f-dabb-4d46-9964-db0fe2dda6fe-proxy-tls podName:111e456f-dabb-4d46-9964-db0fe2dda6fe nodeName:}" failed. No retries permitted until 2026-09-17 17:00:56.009580544 +0000 UTC m=+1121.575692245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/111e456f-dabb-4d46-9964-db0fe2dda6fe-proxy-tls") pod "raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" (UID: "111e456f-dabb-4d46-9964-db0fe2dda6fe") : secret "raw-sklearn-runtime-60bd0-predictor-serving-cert" not found
Sep 17 17:00:55.509895 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.509872    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/111e456f-dabb-4d46-9964-db0fe2dda6fe-kserve-provision-location\") pod \"raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:55.510160 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.510142    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/111e456f-dabb-4d46-9964-db0fe2dda6fe-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:55.510160 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.510147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/111e456f-dabb-4d46-9964-db0fe2dda6fe-cabundle-cert\") pod \"raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:55.518127 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.518106    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdx5\" (UniqueName: \"kubernetes.io/projected/111e456f-dabb-4d46-9964-db0fe2dda6fe-kube-api-access-8zdx5\") pod \"raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:55.630745 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:55.630723    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:00:56.013500 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:56.013420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/111e456f-dabb-4d46-9964-db0fe2dda6fe-proxy-tls\") pod \"raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:56.015863 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:56.015830    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/111e456f-dabb-4d46-9964-db0fe2dda6fe-proxy-tls\") pod \"raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:56.284165 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:56.284104    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:00:56.556074 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:56.556050    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"]
Sep 17 17:00:56.558679 ip-10-0-132-85 kubenswrapper[2565]: W0917 17:00:56.558651    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111e456f_dabb_4d46_9964_db0fe2dda6fe.slice/crio-bd08af53420145d166f53e6fc893e7b61974ab75740b72db6bdd4176cdc957af WatchSource:0}: Error finding container bd08af53420145d166f53e6fc893e7b61974ab75740b72db6bdd4176cdc957af: Status 404 returned error can't find the container with id bd08af53420145d166f53e6fc893e7b61974ab75740b72db6bdd4176cdc957af
Sep 17 17:00:56.589060 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:56.589033    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" event={"ID":"111e456f-dabb-4d46-9964-db0fe2dda6fe","Type":"ContainerStarted","Data":"bd08af53420145d166f53e6fc893e7b61974ab75740b72db6bdd4176cdc957af"}
Sep 17 17:00:57.594417 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:57.594381    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" event={"ID":"111e456f-dabb-4d46-9964-db0fe2dda6fe","Type":"ContainerStarted","Data":"08d162d6e442d93c211ee8067360852d457925b37d9970ff6dad35339902417e"}
Sep 17 17:00:58.819461 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:00:58.819421    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:01:01.611324 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:01.611292    2565 generic.go:356] "Generic (PLEG): container finished" podID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerID="08d162d6e442d93c211ee8067360852d457925b37d9970ff6dad35339902417e" exitCode=0
Sep 17 17:01:01.611774 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:01.611368    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" event={"ID":"111e456f-dabb-4d46-9964-db0fe2dda6fe","Type":"ContainerDied","Data":"08d162d6e442d93c211ee8067360852d457925b37d9970ff6dad35339902417e"}
Sep 17 17:01:02.617281 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:02.617247    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" event={"ID":"111e456f-dabb-4d46-9964-db0fe2dda6fe","Type":"ContainerStarted","Data":"379a40413504252d04c856650dd340fedfe3be90875cb6607863caf2b01acd73"}
Sep 17 17:01:02.617683 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:02.617306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" event={"ID":"111e456f-dabb-4d46-9964-db0fe2dda6fe","Type":"ContainerStarted","Data":"d0bd219e3d56c402d50a6bddaef5e186e4c8cca467d54b200a68d7743770245e"}
Sep 17 17:01:02.617683 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:02.617554    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:01:02.617683 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:02.617601    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:01:02.618860 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:02.618836    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:01:02.637315 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:02.637271    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" podStartSLOduration=3.658016756 podStartE2EDuration="7.637256574s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.979239818s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:01:02.635569794 +0000 UTC m=+1128.201681516" watchObservedRunningTime="2026-09-17 17:01:02.637256574 +0000 UTC m=+1128.203368296"
Sep 17 17:01:03.621839 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:03.621802    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:01:08.627051 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:08.627022    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:01:08.627684 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:08.627654    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:01:08.819731 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:08.819696    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:01:18.628442 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:18.628398    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:01:18.820112 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:18.820066    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:01:28.627765 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:28.627682    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:01:28.820283 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:28.820239    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:01:38.627677 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:38.627637    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:01:38.819731 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:38.819691    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:01:48.627683 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:48.627650    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:01:48.820147 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:48.820107    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:01:58.628359 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:58.628320    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:01:58.823597 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:01:58.823568    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 17:02:02.052710 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:02.052675    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"]
Sep 17 17:02:02.053226 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:02.052955    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" containerID="cri-o://ff29d3f189d5a30a213c2e22cfd33b28a071796270380faa55c37f35370c3598" gracePeriod=30
Sep 17 17:02:02.053226 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:02.053000    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kube-rbac-proxy" containerID="cri-o://414aa749f586f53fd0aa813d808701076dff925be6ae2290c60a12b586efd900" gracePeriod=30
Sep 17 17:02:02.068005 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:02.067946    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:02:02.862000 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:02.861970    2565 generic.go:356] "Generic (PLEG): container finished" podID="6822971d-be41-4680-8fe1-ff883636d593" containerID="414aa749f586f53fd0aa813d808701076dff925be6ae2290c60a12b586efd900" exitCode=2
Sep 17 17:02:02.862168 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:02.862041    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" event={"ID":"6822971d-be41-4680-8fe1-ff883636d593","Type":"ContainerDied","Data":"414aa749f586f53fd0aa813d808701076dff925be6ae2290c60a12b586efd900"}
Sep 17 17:02:04.243399 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:04.243355    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 17:02:07.205173 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.205146    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:02:07.240865 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.240835    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"]
Sep 17 17:02:07.244307 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.244289    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.249528 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.249508    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 17:02:07.249646 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.249541    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 17:02:07.256333 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.256309    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"]
Sep 17 17:02:07.282532 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.282482    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"]
Sep 17 17:02:07.282720 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.282652    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:02:07.286256 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.286240    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:07.288411 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.288392    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 17:02:07.295113 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.295091    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"]
Sep 17 17:02:07.426793 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.426763    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qjtl\" (UniqueName: \"kubernetes.io/projected/62dc5067-5383-4539-8fcd-bc7b85a9015a-kube-api-access-6qjtl\") pod \"isvc-canary-create-predictor-8cb968449-hkr9h\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:07.427005 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.426804    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57661324-5764-4d10-b56a-9d36852c19ac-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7768896644-v24qh\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.427005 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.426833    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57661324-5764-4d10-b56a-9d36852c19ac-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7768896644-v24qh\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.427005 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.426891    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dc5067-5383-4539-8fcd-bc7b85a9015a-cabundle-cert\") pod \"isvc-canary-create-predictor-8cb968449-hkr9h\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:07.427005 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.426944    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwtvr\" (UniqueName: \"kubernetes.io/projected/57661324-5764-4d10-b56a-9d36852c19ac-kube-api-access-cwtvr\") pod \"isvc-canary-create-v2-predictor-7768896644-v24qh\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.427238 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.427011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dc5067-5383-4539-8fcd-bc7b85a9015a-kserve-provision-location\") pod \"isvc-canary-create-predictor-8cb968449-hkr9h\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:07.427238 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.427033    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57661324-5764-4d10-b56a-9d36852c19ac-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7768896644-v24qh\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.427238 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.427057    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62dc5067-5383-4539-8fcd-bc7b85a9015a-proxy-tls\") pod \"isvc-canary-create-predictor-8cb968449-hkr9h\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:07.427238 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.427083    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62dc5067-5383-4539-8fcd-bc7b85a9015a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8cb968449-hkr9h\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:07.427238 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.427134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57661324-5764-4d10-b56a-9d36852c19ac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7768896644-v24qh\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.458616 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.458552    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:02:07.527669 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.527636    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qjtl\" (UniqueName: \"kubernetes.io/projected/62dc5067-5383-4539-8fcd-bc7b85a9015a-kube-api-access-6qjtl\") pod \"isvc-canary-create-predictor-8cb968449-hkr9h\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:07.527798 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.527673    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57661324-5764-4d10-b56a-9d36852c19ac-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7768896644-v24qh\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.527798 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.527702    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57661324-5764-4d10-b56a-9d36852c19ac-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7768896644-v24qh\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.527798 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.527730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dc5067-5383-4539-8fcd-bc7b85a9015a-cabundle-cert\") pod \"isvc-canary-create-predictor-8cb968449-hkr9h\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:07.527798 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.527757    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwtvr\" (UniqueName: \"kubernetes.io/projected/57661324-5764-4d10-b56a-9d36852c19ac-kube-api-access-cwtvr\") pod \"isvc-canary-create-v2-predictor-7768896644-v24qh\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.528058 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.527810    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dc5067-5383-4539-8fcd-bc7b85a9015a-kserve-provision-location\") pod \"isvc-canary-create-predictor-8cb968449-hkr9h\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:07.528058 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.527836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57661324-5764-4d10-b56a-9d36852c19ac-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7768896644-v24qh\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.528058 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.527862    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62dc5067-5383-4539-8fcd-bc7b85a9015a-proxy-tls\") pod \"isvc-canary-create-predictor-8cb968449-hkr9h\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:07.528058 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.527887    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62dc5067-5383-4539-8fcd-bc7b85a9015a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8cb968449-hkr9h\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:07.528058 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.527956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57661324-5764-4d10-b56a-9d36852c19ac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7768896644-v24qh\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.528058 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.528038    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57661324-5764-4d10-b56a-9d36852c19ac-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7768896644-v24qh\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.528365 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.528307    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dc5067-5383-4539-8fcd-bc7b85a9015a-kserve-provision-location\") pod \"isvc-canary-create-predictor-8cb968449-hkr9h\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:07.528428 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:02:07.528414    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 17:02:07.528488 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:02:07.528459    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62dc5067-5383-4539-8fcd-bc7b85a9015a-proxy-tls podName:62dc5067-5383-4539-8fcd-bc7b85a9015a nodeName:}" failed. No retries permitted until 2026-09-17 17:02:08.028445736 +0000 UTC m=+1193.594557436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/62dc5067-5383-4539-8fcd-bc7b85a9015a-proxy-tls") pod "isvc-canary-create-predictor-8cb968449-hkr9h" (UID: "62dc5067-5383-4539-8fcd-bc7b85a9015a") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 17:02:07.528561 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.528524    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57661324-5764-4d10-b56a-9d36852c19ac-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7768896644-v24qh\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.528627 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.528610    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dc5067-5383-4539-8fcd-bc7b85a9015a-cabundle-cert\") pod \"isvc-canary-create-predictor-8cb968449-hkr9h\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:07.528714 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.528694    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62dc5067-5383-4539-8fcd-bc7b85a9015a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8cb968449-hkr9h\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:07.528765 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.528697    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57661324-5764-4d10-b56a-9d36852c19ac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7768896644-v24qh\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.530359 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.530341    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57661324-5764-4d10-b56a-9d36852c19ac-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7768896644-v24qh\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.539602 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.539577    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qjtl\" (UniqueName: \"kubernetes.io/projected/62dc5067-5383-4539-8fcd-bc7b85a9015a-kube-api-access-6qjtl\") pod \"isvc-canary-create-predictor-8cb968449-hkr9h\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:07.539811 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.539795    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwtvr\" (UniqueName: \"kubernetes.io/projected/57661324-5764-4d10-b56a-9d36852c19ac-kube-api-access-cwtvr\") pod \"isvc-canary-create-v2-predictor-7768896644-v24qh\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.556630 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.556611    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:07.573291 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.573265    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:02:07.827446 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.827425    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"]
Sep 17 17:02:07.829607 ip-10-0-132-85 kubenswrapper[2565]: W0917 17:02:07.829576    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57661324_5764_4d10_b56a_9d36852c19ac.slice/crio-47cedc17651a3f2504e78369ff7f5cefd7c36775ba68b18a4e0962d4e0af06fa WatchSource:0}: Error finding container 47cedc17651a3f2504e78369ff7f5cefd7c36775ba68b18a4e0962d4e0af06fa: Status 404 returned error can't find the container with id 47cedc17651a3f2504e78369ff7f5cefd7c36775ba68b18a4e0962d4e0af06fa
Sep 17 17:02:07.882790 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:07.882765    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" event={"ID":"57661324-5764-4d10-b56a-9d36852c19ac","Type":"ContainerStarted","Data":"47cedc17651a3f2504e78369ff7f5cefd7c36775ba68b18a4e0962d4e0af06fa"}
Sep 17 17:02:08.030684 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:08.030651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62dc5067-5383-4539-8fcd-bc7b85a9015a-proxy-tls\") pod \"isvc-canary-create-predictor-8cb968449-hkr9h\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:08.032909 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:08.032882    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62dc5067-5383-4539-8fcd-bc7b85a9015a-proxy-tls\") pod \"isvc-canary-create-predictor-8cb968449-hkr9h\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:08.195564 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:08.195537    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:08.466003 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:08.465970    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"]
Sep 17 17:02:08.468268 ip-10-0-132-85 kubenswrapper[2565]: W0917 17:02:08.468244    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62dc5067_5383_4539_8fcd_bc7b85a9015a.slice/crio-87e65daf7a9f2a818d6f91487f334c89f180ce118fdec5fbdf0c9bc851d697e6 WatchSource:0}: Error finding container 87e65daf7a9f2a818d6f91487f334c89f180ce118fdec5fbdf0c9bc851d697e6: Status 404 returned error can't find the container with id 87e65daf7a9f2a818d6f91487f334c89f180ce118fdec5fbdf0c9bc851d697e6
Sep 17 17:02:08.627553 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:08.627521    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:02:08.820449 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:08.820366    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 17:02:08.888210 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:08.888176    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" event={"ID":"57661324-5764-4d10-b56a-9d36852c19ac","Type":"ContainerStarted","Data":"788ab5cf862ed95f900f6ab0953552973ce335aa2a30a14627fdb1024b72b4b4"}
Sep 17 17:02:08.889785 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:08.889762    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" event={"ID":"62dc5067-5383-4539-8fcd-bc7b85a9015a","Type":"ContainerStarted","Data":"7f3e9ff714cb3d57b4f0bf4165d1c499730c1aa415b5c2700c2c2eb7b8c695fe"}
Sep 17 17:02:08.889900 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:08.889806    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" event={"ID":"62dc5067-5383-4539-8fcd-bc7b85a9015a","Type":"ContainerStarted","Data":"87e65daf7a9f2a818d6f91487f334c89f180ce118fdec5fbdf0c9bc851d697e6"}
Sep 17 17:02:09.243797 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:09.243755    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 17:02:11.089406 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.089383    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 17:02:11.254164 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.254102    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw67n\" (UniqueName: \"kubernetes.io/projected/6822971d-be41-4680-8fe1-ff883636d593-kube-api-access-zw67n\") pod \"6822971d-be41-4680-8fe1-ff883636d593\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") "
Sep 17 17:02:11.254164 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.254144    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6822971d-be41-4680-8fe1-ff883636d593-cabundle-cert\") pod \"6822971d-be41-4680-8fe1-ff883636d593\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") "
Sep 17 17:02:11.254307 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.254182    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6822971d-be41-4680-8fe1-ff883636d593-kube-rbac-proxy-sar-config\") pod \"6822971d-be41-4680-8fe1-ff883636d593\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") "
Sep 17 17:02:11.254307 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.254225    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6822971d-be41-4680-8fe1-ff883636d593-proxy-tls\") pod \"6822971d-be41-4680-8fe1-ff883636d593\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") "
Sep 17 17:02:11.254307 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.254261    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6822971d-be41-4680-8fe1-ff883636d593-kserve-provision-location\") pod \"6822971d-be41-4680-8fe1-ff883636d593\" (UID: \"6822971d-be41-4680-8fe1-ff883636d593\") "
Sep 17 17:02:11.254621 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.254595    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6822971d-be41-4680-8fe1-ff883636d593-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6822971d-be41-4680-8fe1-ff883636d593" (UID: "6822971d-be41-4680-8fe1-ff883636d593"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:02:11.254733 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.254610    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6822971d-be41-4680-8fe1-ff883636d593-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6822971d-be41-4680-8fe1-ff883636d593" (UID: "6822971d-be41-4680-8fe1-ff883636d593"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:02:11.254733 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.254621    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6822971d-be41-4680-8fe1-ff883636d593-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6822971d-be41-4680-8fe1-ff883636d593" (UID: "6822971d-be41-4680-8fe1-ff883636d593"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:11.256152 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.256126    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6822971d-be41-4680-8fe1-ff883636d593-kube-api-access-zw67n" (OuterVolumeSpecName: "kube-api-access-zw67n") pod "6822971d-be41-4680-8fe1-ff883636d593" (UID: "6822971d-be41-4680-8fe1-ff883636d593"). InnerVolumeSpecName "kube-api-access-zw67n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:02:11.256287 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.256263    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6822971d-be41-4680-8fe1-ff883636d593-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6822971d-be41-4680-8fe1-ff883636d593" (UID: "6822971d-be41-4680-8fe1-ff883636d593"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:02:11.355100 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.355069    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6822971d-be41-4680-8fe1-ff883636d593-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:02:11.355100 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.355090    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6822971d-be41-4680-8fe1-ff883636d593-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:02:11.355100 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.355100    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6822971d-be41-4680-8fe1-ff883636d593-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:02:11.355333 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.355109    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6822971d-be41-4680-8fe1-ff883636d593-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:02:11.355333 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.355117    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw67n\" (UniqueName: \"kubernetes.io/projected/6822971d-be41-4680-8fe1-ff883636d593-kube-api-access-zw67n\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:02:11.903087 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.903056    2565 generic.go:356] "Generic (PLEG): container finished" podID="6822971d-be41-4680-8fe1-ff883636d593" containerID="ff29d3f189d5a30a213c2e22cfd33b28a071796270380faa55c37f35370c3598" exitCode=0
Sep 17 17:02:11.903247 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.903113    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"
Sep 17 17:02:11.903247 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.903147    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" event={"ID":"6822971d-be41-4680-8fe1-ff883636d593","Type":"ContainerDied","Data":"ff29d3f189d5a30a213c2e22cfd33b28a071796270380faa55c37f35370c3598"}
Sep 17 17:02:11.903247 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.903206    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g" event={"ID":"6822971d-be41-4680-8fe1-ff883636d593","Type":"ContainerDied","Data":"55f121b97c4f8490eb6766747131bc1f358f6b44e90aa1843d2f3b107b02f834"}
Sep 17 17:02:11.903378 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.903274    2565 scope.go:117] "RemoveContainer" containerID="414aa749f586f53fd0aa813d808701076dff925be6ae2290c60a12b586efd900"
Sep 17 17:02:11.904870 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.904835    2565 generic.go:356] "Generic (PLEG): container finished" podID="57661324-5764-4d10-b56a-9d36852c19ac" containerID="788ab5cf862ed95f900f6ab0953552973ce335aa2a30a14627fdb1024b72b4b4" exitCode=0
Sep 17 17:02:11.905071 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.904888    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" event={"ID":"57661324-5764-4d10-b56a-9d36852c19ac","Type":"ContainerDied","Data":"788ab5cf862ed95f900f6ab0953552973ce335aa2a30a14627fdb1024b72b4b4"}
Sep 17 17:02:11.914040 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.914020    2565 scope.go:117] "RemoveContainer" containerID="ff29d3f189d5a30a213c2e22cfd33b28a071796270380faa55c37f35370c3598"
Sep 17 17:02:11.922090 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.922068    2565 scope.go:117] "RemoveContainer" containerID="23151ecbf6f2b1704c5c125ee2bc31f29c334ff0e708153d23fccc5efb8e23b9"
Sep 17 17:02:11.931112 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.931091    2565 scope.go:117] "RemoveContainer" containerID="414aa749f586f53fd0aa813d808701076dff925be6ae2290c60a12b586efd900"
Sep 17 17:02:11.931544 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:02:11.931400    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414aa749f586f53fd0aa813d808701076dff925be6ae2290c60a12b586efd900\": container with ID starting with 414aa749f586f53fd0aa813d808701076dff925be6ae2290c60a12b586efd900 not found: ID does not exist" containerID="414aa749f586f53fd0aa813d808701076dff925be6ae2290c60a12b586efd900"
Sep 17 17:02:11.931544 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.931439    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414aa749f586f53fd0aa813d808701076dff925be6ae2290c60a12b586efd900"} err="failed to get container status \"414aa749f586f53fd0aa813d808701076dff925be6ae2290c60a12b586efd900\": rpc error: code = NotFound desc = could not find container \"414aa749f586f53fd0aa813d808701076dff925be6ae2290c60a12b586efd900\": container with ID starting with 414aa749f586f53fd0aa813d808701076dff925be6ae2290c60a12b586efd900 not found: ID does not exist"
Sep 17 17:02:11.931544 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.931463    2565 scope.go:117] "RemoveContainer" containerID="ff29d3f189d5a30a213c2e22cfd33b28a071796270380faa55c37f35370c3598"
Sep 17 17:02:11.931761 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:02:11.931741    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff29d3f189d5a30a213c2e22cfd33b28a071796270380faa55c37f35370c3598\": container with ID starting with ff29d3f189d5a30a213c2e22cfd33b28a071796270380faa55c37f35370c3598 not found: ID does not exist" containerID="ff29d3f189d5a30a213c2e22cfd33b28a071796270380faa55c37f35370c3598"
Sep 17 17:02:11.931804 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.931766    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff29d3f189d5a30a213c2e22cfd33b28a071796270380faa55c37f35370c3598"} err="failed to get container status \"ff29d3f189d5a30a213c2e22cfd33b28a071796270380faa55c37f35370c3598\": rpc error: code = NotFound desc = could not find container \"ff29d3f189d5a30a213c2e22cfd33b28a071796270380faa55c37f35370c3598\": container with ID starting with ff29d3f189d5a30a213c2e22cfd33b28a071796270380faa55c37f35370c3598 not found: ID does not exist"
Sep 17 17:02:11.931804 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.931781    2565 scope.go:117] "RemoveContainer" containerID="23151ecbf6f2b1704c5c125ee2bc31f29c334ff0e708153d23fccc5efb8e23b9"
Sep 17 17:02:11.932079 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:02:11.932061    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23151ecbf6f2b1704c5c125ee2bc31f29c334ff0e708153d23fccc5efb8e23b9\": container with ID starting with 23151ecbf6f2b1704c5c125ee2bc31f29c334ff0e708153d23fccc5efb8e23b9 not found: ID does not exist" containerID="23151ecbf6f2b1704c5c125ee2bc31f29c334ff0e708153d23fccc5efb8e23b9"
Sep 17 17:02:11.932139 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.932083    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23151ecbf6f2b1704c5c125ee2bc31f29c334ff0e708153d23fccc5efb8e23b9"} err="failed to get container status \"23151ecbf6f2b1704c5c125ee2bc31f29c334ff0e708153d23fccc5efb8e23b9\": rpc error: code = NotFound desc = could not find container \"23151ecbf6f2b1704c5c125ee2bc31f29c334ff0e708153d23fccc5efb8e23b9\": container with ID starting with 23151ecbf6f2b1704c5c125ee2bc31f29c334ff0e708153d23fccc5efb8e23b9 not found: ID does not exist"
Sep 17 17:02:11.940810 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.940785    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"]
Sep 17 17:02:11.946656 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:11.946635    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f447b-predictor-9df674fc8-hp49g"]
Sep 17 17:02:12.823702 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:12.823634    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6822971d-be41-4680-8fe1-ff883636d593" path="/var/lib/kubelet/pods/6822971d-be41-4680-8fe1-ff883636d593/volumes"
Sep 17 17:02:12.910831 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:12.910795    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" event={"ID":"57661324-5764-4d10-b56a-9d36852c19ac","Type":"ContainerStarted","Data":"095193125caa5a6ad2e05d0f13c18052f30c7c973cde98d4030d880fd9b5ad84"}
Sep 17 17:02:12.910973 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:12.910867    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" event={"ID":"57661324-5764-4d10-b56a-9d36852c19ac","Type":"ContainerStarted","Data":"5f867e404f8d5255ede1a9d0554e9c4c23e3841c1e93b0a432089e2f9e1f14d4"}
Sep 17 17:02:12.911280 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:12.911255    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:12.911403 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:12.911312    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:12.912457 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:12.912431    2565 generic.go:356] "Generic (PLEG): container finished" podID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerID="7f3e9ff714cb3d57b4f0bf4165d1c499730c1aa415b5c2700c2c2eb7b8c695fe" exitCode=0
Sep 17 17:02:12.912553 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:12.912497    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" event={"ID":"62dc5067-5383-4539-8fcd-bc7b85a9015a","Type":"ContainerDied","Data":"7f3e9ff714cb3d57b4f0bf4165d1c499730c1aa415b5c2700c2c2eb7b8c695fe"}
Sep 17 17:02:12.912749 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:12.912727    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:02:12.931508 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:12.931468    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" podStartSLOduration=2.216782256 podStartE2EDuration="5.931453988s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.714671732s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:02:12.930198069 +0000 UTC m=+1198.496309808" watchObservedRunningTime="2026-09-17 17:02:12.931453988 +0000 UTC m=+1198.497565710"
Sep 17 17:02:13.918395 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:13.918360    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" event={"ID":"62dc5067-5383-4539-8fcd-bc7b85a9015a","Type":"ContainerStarted","Data":"798ef2e5ba3615dd3e4a30fa950ceba7ed6d60f71087f370317cd3d33927b597"}
Sep 17 17:02:13.918800 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:13.918433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" event={"ID":"62dc5067-5383-4539-8fcd-bc7b85a9015a","Type":"ContainerStarted","Data":"4457018d767b0ad994393766518f17191ac7949e00af45afb18e14d608e18fbb"}
Sep 17 17:02:13.918800 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:13.918658    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:02:13.918955 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:13.918808    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:13.918955 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:13.918858    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:13.920314 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:13.920285    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:02:13.938290 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:13.938238    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" podStartSLOduration=2.952696055 podStartE2EDuration="6.938220869s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.985524814s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:02:13.935636985 +0000 UTC m=+1199.501748707" watchObservedRunningTime="2026-09-17 17:02:13.938220869 +0000 UTC m=+1199.504332592"
Sep 17 17:02:14.922228 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:14.922185    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:02:18.628133 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:18.628104    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:02:18.923815 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:18.923742    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:02:18.924316 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:18.924290    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:02:19.927044 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:19.927015    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:02:19.927686 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:19.927658    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:02:25.400989 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:25.400946    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"]
Sep 17 17:02:25.401397 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:25.401190    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kserve-container" containerID="cri-o://d0bd219e3d56c402d50a6bddaef5e186e4c8cca467d54b200a68d7743770245e" gracePeriod=30
Sep 17 17:02:25.401397 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:25.401224    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kube-rbac-proxy" containerID="cri-o://379a40413504252d04c856650dd340fedfe3be90875cb6607863caf2b01acd73" gracePeriod=30
Sep 17 17:02:25.428790 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:25.428769    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:02:25.968184 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:25.968150    2565 generic.go:356] "Generic (PLEG): container finished" podID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerID="379a40413504252d04c856650dd340fedfe3be90875cb6607863caf2b01acd73" exitCode=2
Sep 17 17:02:25.968355 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:25.968232    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" event={"ID":"111e456f-dabb-4d46-9964-db0fe2dda6fe","Type":"ContainerDied","Data":"379a40413504252d04c856650dd340fedfe3be90875cb6607863caf2b01acd73"}
Sep 17 17:02:28.622199 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:28.622153    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 17 17:02:28.628110 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:28.628084    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 17:02:28.925079 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:28.924996    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:02:29.650990 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.650967    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:02:29.695075 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.695049    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/111e456f-dabb-4d46-9964-db0fe2dda6fe-proxy-tls\") pod \"111e456f-dabb-4d46-9964-db0fe2dda6fe\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") "
Sep 17 17:02:29.695220 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.695085    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zdx5\" (UniqueName: \"kubernetes.io/projected/111e456f-dabb-4d46-9964-db0fe2dda6fe-kube-api-access-8zdx5\") pod \"111e456f-dabb-4d46-9964-db0fe2dda6fe\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") "
Sep 17 17:02:29.695220 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.695135    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/111e456f-dabb-4d46-9964-db0fe2dda6fe-cabundle-cert\") pod \"111e456f-dabb-4d46-9964-db0fe2dda6fe\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") "
Sep 17 17:02:29.695220 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.695176    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/111e456f-dabb-4d46-9964-db0fe2dda6fe-kserve-provision-location\") pod \"111e456f-dabb-4d46-9964-db0fe2dda6fe\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") "
Sep 17 17:02:29.695386 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.695263    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/111e456f-dabb-4d46-9964-db0fe2dda6fe-kube-rbac-proxy-sar-config\") pod \"111e456f-dabb-4d46-9964-db0fe2dda6fe\" (UID: \"111e456f-dabb-4d46-9964-db0fe2dda6fe\") "
Sep 17 17:02:29.695566 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.695534    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111e456f-dabb-4d46-9964-db0fe2dda6fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "111e456f-dabb-4d46-9964-db0fe2dda6fe" (UID: "111e456f-dabb-4d46-9964-db0fe2dda6fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:02:29.695566 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.695481    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111e456f-dabb-4d46-9964-db0fe2dda6fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "111e456f-dabb-4d46-9964-db0fe2dda6fe" (UID: "111e456f-dabb-4d46-9964-db0fe2dda6fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:02:29.695732 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.695635    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111e456f-dabb-4d46-9964-db0fe2dda6fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "111e456f-dabb-4d46-9964-db0fe2dda6fe" (UID: "111e456f-dabb-4d46-9964-db0fe2dda6fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:02:29.697110 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.697063    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/111e456f-dabb-4d46-9964-db0fe2dda6fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "111e456f-dabb-4d46-9964-db0fe2dda6fe" (UID: "111e456f-dabb-4d46-9964-db0fe2dda6fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:02:29.697207 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.697187    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111e456f-dabb-4d46-9964-db0fe2dda6fe-kube-api-access-8zdx5" (OuterVolumeSpecName: "kube-api-access-8zdx5") pod "111e456f-dabb-4d46-9964-db0fe2dda6fe" (UID: "111e456f-dabb-4d46-9964-db0fe2dda6fe"). InnerVolumeSpecName "kube-api-access-8zdx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:02:29.795943 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.795907    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/111e456f-dabb-4d46-9964-db0fe2dda6fe-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:02:29.795943 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.795943    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/111e456f-dabb-4d46-9964-db0fe2dda6fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:02:29.796088 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.795952    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/111e456f-dabb-4d46-9964-db0fe2dda6fe-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:02:29.796088 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.795962    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zdx5\" (UniqueName: \"kubernetes.io/projected/111e456f-dabb-4d46-9964-db0fe2dda6fe-kube-api-access-8zdx5\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:02:29.796088 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.795971    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/111e456f-dabb-4d46-9964-db0fe2dda6fe-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:02:29.927979 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.927951    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:02:29.985237 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.985154    2565 generic.go:356] "Generic (PLEG): container finished" podID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerID="d0bd219e3d56c402d50a6bddaef5e186e4c8cca467d54b200a68d7743770245e" exitCode=0
Sep 17 17:02:29.985237 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.985216    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"
Sep 17 17:02:29.985438 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.985244    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" event={"ID":"111e456f-dabb-4d46-9964-db0fe2dda6fe","Type":"ContainerDied","Data":"d0bd219e3d56c402d50a6bddaef5e186e4c8cca467d54b200a68d7743770245e"}
Sep 17 17:02:29.985438 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.985320    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2" event={"ID":"111e456f-dabb-4d46-9964-db0fe2dda6fe","Type":"ContainerDied","Data":"bd08af53420145d166f53e6fc893e7b61974ab75740b72db6bdd4176cdc957af"}
Sep 17 17:02:29.985438 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.985366    2565 scope.go:117] "RemoveContainer" containerID="379a40413504252d04c856650dd340fedfe3be90875cb6607863caf2b01acd73"
Sep 17 17:02:29.994504 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:29.994487    2565 scope.go:117] "RemoveContainer" containerID="d0bd219e3d56c402d50a6bddaef5e186e4c8cca467d54b200a68d7743770245e"
Sep 17 17:02:30.001508 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:30.001491    2565 scope.go:117] "RemoveContainer" containerID="08d162d6e442d93c211ee8067360852d457925b37d9970ff6dad35339902417e"
Sep 17 17:02:30.007161 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:30.007140    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"]
Sep 17 17:02:30.009487 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:30.009471    2565 scope.go:117] "RemoveContainer" containerID="379a40413504252d04c856650dd340fedfe3be90875cb6607863caf2b01acd73"
Sep 17 17:02:30.009754 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:02:30.009737    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379a40413504252d04c856650dd340fedfe3be90875cb6607863caf2b01acd73\": container with ID starting with 379a40413504252d04c856650dd340fedfe3be90875cb6607863caf2b01acd73 not found: ID does not exist" containerID="379a40413504252d04c856650dd340fedfe3be90875cb6607863caf2b01acd73"
Sep 17 17:02:30.009802 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:30.009762    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379a40413504252d04c856650dd340fedfe3be90875cb6607863caf2b01acd73"} err="failed to get container status \"379a40413504252d04c856650dd340fedfe3be90875cb6607863caf2b01acd73\": rpc error: code = NotFound desc = could not find container \"379a40413504252d04c856650dd340fedfe3be90875cb6607863caf2b01acd73\": container with ID starting with 379a40413504252d04c856650dd340fedfe3be90875cb6607863caf2b01acd73 not found: ID does not exist"
Sep 17 17:02:30.009802 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:30.009778    2565 scope.go:117] "RemoveContainer" containerID="d0bd219e3d56c402d50a6bddaef5e186e4c8cca467d54b200a68d7743770245e"
Sep 17 17:02:30.010060 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:02:30.010042    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0bd219e3d56c402d50a6bddaef5e186e4c8cca467d54b200a68d7743770245e\": container with ID starting with d0bd219e3d56c402d50a6bddaef5e186e4c8cca467d54b200a68d7743770245e not found: ID does not exist" containerID="d0bd219e3d56c402d50a6bddaef5e186e4c8cca467d54b200a68d7743770245e"
Sep 17 17:02:30.010112 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:30.010067    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0bd219e3d56c402d50a6bddaef5e186e4c8cca467d54b200a68d7743770245e"} err="failed to get container status \"d0bd219e3d56c402d50a6bddaef5e186e4c8cca467d54b200a68d7743770245e\": rpc error: code = NotFound desc = could not find container \"d0bd219e3d56c402d50a6bddaef5e186e4c8cca467d54b200a68d7743770245e\": container with ID starting with d0bd219e3d56c402d50a6bddaef5e186e4c8cca467d54b200a68d7743770245e not found: ID does not exist"
Sep 17 17:02:30.010112 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:30.010082    2565 scope.go:117] "RemoveContainer" containerID="08d162d6e442d93c211ee8067360852d457925b37d9970ff6dad35339902417e"
Sep 17 17:02:30.010325 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:02:30.010308    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d162d6e442d93c211ee8067360852d457925b37d9970ff6dad35339902417e\": container with ID starting with 08d162d6e442d93c211ee8067360852d457925b37d9970ff6dad35339902417e not found: ID does not exist" containerID="08d162d6e442d93c211ee8067360852d457925b37d9970ff6dad35339902417e"
Sep 17 17:02:30.010368 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:30.010332    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d162d6e442d93c211ee8067360852d457925b37d9970ff6dad35339902417e"} err="failed to get container status \"08d162d6e442d93c211ee8067360852d457925b37d9970ff6dad35339902417e\": rpc error: code = NotFound desc = could not find container \"08d162d6e442d93c211ee8067360852d457925b37d9970ff6dad35339902417e\": container with ID starting with 08d162d6e442d93c211ee8067360852d457925b37d9970ff6dad35339902417e not found: ID does not exist"
Sep 17 17:02:30.013502 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:30.013478    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-60bd0-predictor-7597559d5b-wcdl2"]
Sep 17 17:02:30.823313 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:30.823275    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" path="/var/lib/kubelet/pods/111e456f-dabb-4d46-9964-db0fe2dda6fe/volumes"
Sep 17 17:02:38.924311 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:38.924273    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:02:39.928187 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:39.928152    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:02:48.924563 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:48.924520    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:02:49.927988 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:49.927952    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:02:58.925139 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:58.925060    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:02:59.927501 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:02:59.927461    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:03:08.924927 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:08.924872    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:03:09.927760 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:09.927722    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:03:18.925071 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:18.925025    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:03:19.927841 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:19.927795    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 17:03:28.925518 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:28.925487    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:03:29.049538 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:29.049508    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"]
Sep 17 17:03:29.049746 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:29.049717    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kserve-container" containerID="cri-o://4457018d767b0ad994393766518f17191ac7949e00af45afb18e14d608e18fbb" gracePeriod=30
Sep 17 17:03:29.049880 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:29.049791    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kube-rbac-proxy" containerID="cri-o://798ef2e5ba3615dd3e4a30fa950ceba7ed6d60f71087f370317cd3d33927b597" gracePeriod=30
Sep 17 17:03:29.050688 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:29.050661    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:03:29.215260 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:29.215176    2565 generic.go:356] "Generic (PLEG): container finished" podID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerID="798ef2e5ba3615dd3e4a30fa950ceba7ed6d60f71087f370317cd3d33927b597" exitCode=2
Sep 17 17:03:29.215410 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:29.215252    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" event={"ID":"62dc5067-5383-4539-8fcd-bc7b85a9015a","Type":"ContainerDied","Data":"798ef2e5ba3615dd3e4a30fa950ceba7ed6d60f71087f370317cd3d33927b597"}
Sep 17 17:03:29.922760 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:29.922713    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 17:03:32.896892 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:32.896870    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:03:32.953364 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:32.953307    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qjtl\" (UniqueName: \"kubernetes.io/projected/62dc5067-5383-4539-8fcd-bc7b85a9015a-kube-api-access-6qjtl\") pod \"62dc5067-5383-4539-8fcd-bc7b85a9015a\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") "
Sep 17 17:03:32.953364 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:32.953352    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dc5067-5383-4539-8fcd-bc7b85a9015a-kserve-provision-location\") pod \"62dc5067-5383-4539-8fcd-bc7b85a9015a\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") "
Sep 17 17:03:32.953557 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:32.953393    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62dc5067-5383-4539-8fcd-bc7b85a9015a-kube-rbac-proxy-sar-config\") pod \"62dc5067-5383-4539-8fcd-bc7b85a9015a\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") "
Sep 17 17:03:32.953557 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:32.953426    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62dc5067-5383-4539-8fcd-bc7b85a9015a-proxy-tls\") pod \"62dc5067-5383-4539-8fcd-bc7b85a9015a\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") "
Sep 17 17:03:32.953557 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:32.953447    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dc5067-5383-4539-8fcd-bc7b85a9015a-cabundle-cert\") pod \"62dc5067-5383-4539-8fcd-bc7b85a9015a\" (UID: \"62dc5067-5383-4539-8fcd-bc7b85a9015a\") "
Sep 17 17:03:32.953767 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:32.953735    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dc5067-5383-4539-8fcd-bc7b85a9015a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62dc5067-5383-4539-8fcd-bc7b85a9015a" (UID: "62dc5067-5383-4539-8fcd-bc7b85a9015a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:03:32.953845 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:32.953789    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62dc5067-5383-4539-8fcd-bc7b85a9015a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "62dc5067-5383-4539-8fcd-bc7b85a9015a" (UID: "62dc5067-5383-4539-8fcd-bc7b85a9015a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:03:32.953908 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:32.953880    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62dc5067-5383-4539-8fcd-bc7b85a9015a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "62dc5067-5383-4539-8fcd-bc7b85a9015a" (UID: "62dc5067-5383-4539-8fcd-bc7b85a9015a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:03:32.955413 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:32.955395    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62dc5067-5383-4539-8fcd-bc7b85a9015a-kube-api-access-6qjtl" (OuterVolumeSpecName: "kube-api-access-6qjtl") pod "62dc5067-5383-4539-8fcd-bc7b85a9015a" (UID: "62dc5067-5383-4539-8fcd-bc7b85a9015a"). InnerVolumeSpecName "kube-api-access-6qjtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:03:32.955773 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:32.955750    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62dc5067-5383-4539-8fcd-bc7b85a9015a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "62dc5067-5383-4539-8fcd-bc7b85a9015a" (UID: "62dc5067-5383-4539-8fcd-bc7b85a9015a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:03:33.054307 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.054276    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dc5067-5383-4539-8fcd-bc7b85a9015a-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:03:33.054307 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.054306    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qjtl\" (UniqueName: \"kubernetes.io/projected/62dc5067-5383-4539-8fcd-bc7b85a9015a-kube-api-access-6qjtl\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:03:33.054307 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.054317    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dc5067-5383-4539-8fcd-bc7b85a9015a-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:03:33.054492 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.054325    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62dc5067-5383-4539-8fcd-bc7b85a9015a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:03:33.054492 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.054334    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62dc5067-5383-4539-8fcd-bc7b85a9015a-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:03:33.232569 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.232499    2565 generic.go:356] "Generic (PLEG): container finished" podID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerID="4457018d767b0ad994393766518f17191ac7949e00af45afb18e14d608e18fbb" exitCode=0
Sep 17 17:03:33.232569 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.232565    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"
Sep 17 17:03:33.232714 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.232596    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" event={"ID":"62dc5067-5383-4539-8fcd-bc7b85a9015a","Type":"ContainerDied","Data":"4457018d767b0ad994393766518f17191ac7949e00af45afb18e14d608e18fbb"}
Sep 17 17:03:33.232714 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.232657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h" event={"ID":"62dc5067-5383-4539-8fcd-bc7b85a9015a","Type":"ContainerDied","Data":"87e65daf7a9f2a818d6f91487f334c89f180ce118fdec5fbdf0c9bc851d697e6"}
Sep 17 17:03:33.232714 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.232693    2565 scope.go:117] "RemoveContainer" containerID="798ef2e5ba3615dd3e4a30fa950ceba7ed6d60f71087f370317cd3d33927b597"
Sep 17 17:03:33.243390 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.243371    2565 scope.go:117] "RemoveContainer" containerID="4457018d767b0ad994393766518f17191ac7949e00af45afb18e14d608e18fbb"
Sep 17 17:03:33.250626 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.250610    2565 scope.go:117] "RemoveContainer" containerID="7f3e9ff714cb3d57b4f0bf4165d1c499730c1aa415b5c2700c2c2eb7b8c695fe"
Sep 17 17:03:33.257291 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.257272    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"]
Sep 17 17:03:33.258222 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.258208    2565 scope.go:117] "RemoveContainer" containerID="798ef2e5ba3615dd3e4a30fa950ceba7ed6d60f71087f370317cd3d33927b597"
Sep 17 17:03:33.258502 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:03:33.258482    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798ef2e5ba3615dd3e4a30fa950ceba7ed6d60f71087f370317cd3d33927b597\": container with ID starting with 798ef2e5ba3615dd3e4a30fa950ceba7ed6d60f71087f370317cd3d33927b597 not found: ID does not exist" containerID="798ef2e5ba3615dd3e4a30fa950ceba7ed6d60f71087f370317cd3d33927b597"
Sep 17 17:03:33.258596 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.258509    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798ef2e5ba3615dd3e4a30fa950ceba7ed6d60f71087f370317cd3d33927b597"} err="failed to get container status \"798ef2e5ba3615dd3e4a30fa950ceba7ed6d60f71087f370317cd3d33927b597\": rpc error: code = NotFound desc = could not find container \"798ef2e5ba3615dd3e4a30fa950ceba7ed6d60f71087f370317cd3d33927b597\": container with ID starting with 798ef2e5ba3615dd3e4a30fa950ceba7ed6d60f71087f370317cd3d33927b597 not found: ID does not exist"
Sep 17 17:03:33.258596 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.258526    2565 scope.go:117] "RemoveContainer" containerID="4457018d767b0ad994393766518f17191ac7949e00af45afb18e14d608e18fbb"
Sep 17 17:03:33.258778 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:03:33.258761    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4457018d767b0ad994393766518f17191ac7949e00af45afb18e14d608e18fbb\": container with ID starting with 4457018d767b0ad994393766518f17191ac7949e00af45afb18e14d608e18fbb not found: ID does not exist" containerID="4457018d767b0ad994393766518f17191ac7949e00af45afb18e14d608e18fbb"
Sep 17 17:03:33.258841 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.258782    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4457018d767b0ad994393766518f17191ac7949e00af45afb18e14d608e18fbb"} err="failed to get container status \"4457018d767b0ad994393766518f17191ac7949e00af45afb18e14d608e18fbb\": rpc error: code = NotFound desc = could not find container \"4457018d767b0ad994393766518f17191ac7949e00af45afb18e14d608e18fbb\": container with ID starting with 4457018d767b0ad994393766518f17191ac7949e00af45afb18e14d608e18fbb not found: ID does not exist"
Sep 17 17:03:33.258841 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.258794    2565 scope.go:117] "RemoveContainer" containerID="7f3e9ff714cb3d57b4f0bf4165d1c499730c1aa415b5c2700c2c2eb7b8c695fe"
Sep 17 17:03:33.259099 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:03:33.259068    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3e9ff714cb3d57b4f0bf4165d1c499730c1aa415b5c2700c2c2eb7b8c695fe\": container with ID starting with 7f3e9ff714cb3d57b4f0bf4165d1c499730c1aa415b5c2700c2c2eb7b8c695fe not found: ID does not exist" containerID="7f3e9ff714cb3d57b4f0bf4165d1c499730c1aa415b5c2700c2c2eb7b8c695fe"
Sep 17 17:03:33.259192 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.259100    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3e9ff714cb3d57b4f0bf4165d1c499730c1aa415b5c2700c2c2eb7b8c695fe"} err="failed to get container status \"7f3e9ff714cb3d57b4f0bf4165d1c499730c1aa415b5c2700c2c2eb7b8c695fe\": rpc error: code = NotFound desc = could not find container \"7f3e9ff714cb3d57b4f0bf4165d1c499730c1aa415b5c2700c2c2eb7b8c695fe\": container with ID starting with 7f3e9ff714cb3d57b4f0bf4165d1c499730c1aa415b5c2700c2c2eb7b8c695fe not found: ID does not exist"
Sep 17 17:03:33.261433 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:33.261410    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8cb968449-hkr9h"]
Sep 17 17:03:34.823063 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:34.823033    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" path="/var/lib/kubelet/pods/62dc5067-5383-4539-8fcd-bc7b85a9015a/volumes"
Sep 17 17:03:37.319800 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:37.319769    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"]
Sep 17 17:03:37.320245 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:37.320022    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kserve-container" containerID="cri-o://5f867e404f8d5255ede1a9d0554e9c4c23e3841c1e93b0a432089e2f9e1f14d4" gracePeriod=30
Sep 17 17:03:37.320245 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:37.320066    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kube-rbac-proxy" containerID="cri-o://095193125caa5a6ad2e05d0f13c18052f30c7c973cde98d4030d880fd9b5ad84" gracePeriod=30
Sep 17 17:03:37.328865 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:37.328844    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:03:37.428285 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:37.428255    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:03:38.255285 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:38.255249    2565 generic.go:356] "Generic (PLEG): container finished" podID="57661324-5764-4d10-b56a-9d36852c19ac" containerID="095193125caa5a6ad2e05d0f13c18052f30c7c973cde98d4030d880fd9b5ad84" exitCode=2
Sep 17 17:03:38.255471 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:38.255323    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" event={"ID":"57661324-5764-4d10-b56a-9d36852c19ac","Type":"ContainerDied","Data":"095193125caa5a6ad2e05d0f13c18052f30c7c973cde98d4030d880fd9b5ad84"}
Sep 17 17:03:38.918851 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:38.918815    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 17:03:38.925117 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:38.925093    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 17:03:41.269840 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.269809    2565 generic.go:356] "Generic (PLEG): container finished" podID="57661324-5764-4d10-b56a-9d36852c19ac" containerID="5f867e404f8d5255ede1a9d0554e9c4c23e3841c1e93b0a432089e2f9e1f14d4" exitCode=0
Sep 17 17:03:41.270230 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.269883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" event={"ID":"57661324-5764-4d10-b56a-9d36852c19ac","Type":"ContainerDied","Data":"5f867e404f8d5255ede1a9d0554e9c4c23e3841c1e93b0a432089e2f9e1f14d4"}
Sep 17 17:03:41.351748 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.351726    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:03:41.413979 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.413905    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57661324-5764-4d10-b56a-9d36852c19ac-cabundle-cert\") pod \"57661324-5764-4d10-b56a-9d36852c19ac\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") "
Sep 17 17:03:41.413979 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.413963    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57661324-5764-4d10-b56a-9d36852c19ac-kserve-provision-location\") pod \"57661324-5764-4d10-b56a-9d36852c19ac\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") "
Sep 17 17:03:41.414112 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.414030    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57661324-5764-4d10-b56a-9d36852c19ac-proxy-tls\") pod \"57661324-5764-4d10-b56a-9d36852c19ac\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") "
Sep 17 17:03:41.414112 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.414066    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57661324-5764-4d10-b56a-9d36852c19ac-kube-rbac-proxy-sar-config\") pod \"57661324-5764-4d10-b56a-9d36852c19ac\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") "
Sep 17 17:03:41.414112 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.414098    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwtvr\" (UniqueName: \"kubernetes.io/projected/57661324-5764-4d10-b56a-9d36852c19ac-kube-api-access-cwtvr\") pod \"57661324-5764-4d10-b56a-9d36852c19ac\" (UID: \"57661324-5764-4d10-b56a-9d36852c19ac\") "
Sep 17 17:03:41.414255 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.414220    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57661324-5764-4d10-b56a-9d36852c19ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57661324-5764-4d10-b56a-9d36852c19ac" (UID: "57661324-5764-4d10-b56a-9d36852c19ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:03:41.414313 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.414277    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57661324-5764-4d10-b56a-9d36852c19ac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "57661324-5764-4d10-b56a-9d36852c19ac" (UID: "57661324-5764-4d10-b56a-9d36852c19ac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:03:41.414398 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.414381    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57661324-5764-4d10-b56a-9d36852c19ac-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:03:41.414464 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.414400    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57661324-5764-4d10-b56a-9d36852c19ac-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:03:41.414464 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.414411    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57661324-5764-4d10-b56a-9d36852c19ac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "57661324-5764-4d10-b56a-9d36852c19ac" (UID: "57661324-5764-4d10-b56a-9d36852c19ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:03:41.416008 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.415987    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57661324-5764-4d10-b56a-9d36852c19ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "57661324-5764-4d10-b56a-9d36852c19ac" (UID: "57661324-5764-4d10-b56a-9d36852c19ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:03:41.416103 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.416084    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57661324-5764-4d10-b56a-9d36852c19ac-kube-api-access-cwtvr" (OuterVolumeSpecName: "kube-api-access-cwtvr") pod "57661324-5764-4d10-b56a-9d36852c19ac" (UID: "57661324-5764-4d10-b56a-9d36852c19ac"). InnerVolumeSpecName "kube-api-access-cwtvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:03:41.515076 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.515050    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57661324-5764-4d10-b56a-9d36852c19ac-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:03:41.515173 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.515079    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57661324-5764-4d10-b56a-9d36852c19ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:03:41.515173 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:41.515092    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwtvr\" (UniqueName: \"kubernetes.io/projected/57661324-5764-4d10-b56a-9d36852c19ac-kube-api-access-cwtvr\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:03:42.275612 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.275579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh" event={"ID":"57661324-5764-4d10-b56a-9d36852c19ac","Type":"ContainerDied","Data":"47cedc17651a3f2504e78369ff7f5cefd7c36775ba68b18a4e0962d4e0af06fa"}
Sep 17 17:03:42.275612 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.275618    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"
Sep 17 17:03:42.276055 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.275658    2565 scope.go:117] "RemoveContainer" containerID="095193125caa5a6ad2e05d0f13c18052f30c7c973cde98d4030d880fd9b5ad84"
Sep 17 17:03:42.284103 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.284084    2565 scope.go:117] "RemoveContainer" containerID="5f867e404f8d5255ede1a9d0554e9c4c23e3841c1e93b0a432089e2f9e1f14d4"
Sep 17 17:03:42.290813 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.290796    2565 scope.go:117] "RemoveContainer" containerID="788ab5cf862ed95f900f6ab0953552973ce335aa2a30a14627fdb1024b72b4b4"
Sep 17 17:03:42.297164 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.297140    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"]
Sep 17 17:03:42.303140 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.303119    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7768896644-v24qh"]
Sep 17 17:03:42.430485 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.430456    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:03:42.485188 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485161    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"]
Sep 17 17:03:42.485471 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485458    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kube-rbac-proxy"
Sep 17 17:03:42.485515 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485473    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kube-rbac-proxy"
Sep 17 17:03:42.485515 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485481    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kserve-container"
Sep 17 17:03:42.485515 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485486    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kserve-container"
Sep 17 17:03:42.485515 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485497    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="storage-initializer"
Sep 17 17:03:42.485515 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485503    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="storage-initializer"
Sep 17 17:03:42.485515 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485510    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kserve-container"
Sep 17 17:03:42.485515 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485515    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kserve-container"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485522    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kserve-container"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485528    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kserve-container"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485536    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485541    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485548    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kube-rbac-proxy"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485553    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kube-rbac-proxy"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485561    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kube-rbac-proxy"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485568    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kube-rbac-proxy"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485574    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kube-rbac-proxy"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485579    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kube-rbac-proxy"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485584    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="storage-initializer"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485591    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="storage-initializer"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485601    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="storage-initializer"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485606    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="storage-initializer"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485611    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="storage-initializer"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485616    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="storage-initializer"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485658    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kube-rbac-proxy"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485664    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6822971d-be41-4680-8fe1-ff883636d593" containerName="kserve-container"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485670    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kserve-container"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485676    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="57661324-5764-4d10-b56a-9d36852c19ac" containerName="kube-rbac-proxy"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485682    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kube-rbac-proxy"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485688    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kserve-container"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485692    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62dc5067-5383-4539-8fcd-bc7b85a9015a" containerName="kserve-container"
Sep 17 17:03:42.485728 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.485698    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="111e456f-dabb-4d46-9964-db0fe2dda6fe" containerName="kube-rbac-proxy"
Sep 17 17:03:42.490189 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.490169    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:42.492638 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.492615    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"]
Sep 17 17:03:42.492759 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.492737    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 17:03:42.492828 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.492754    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:03:42.493006 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.492907    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlgx8\""
Sep 17 17:03:42.493062 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.493044    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:03:42.493168 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.493150    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 17:03:42.493238 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.493176    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:03:42.493291 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.493266    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:03:42.496051 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.496035    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.498407 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.498388    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 17:03:42.499256 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.499239    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"]
Sep 17 17:03:42.508396 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.508370    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"]
Sep 17 17:03:42.514526 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.514506    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:03:42.521546 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.521528    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a118f509-5afa-4920-9b48-42b7966034fb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-2gm85\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:42.521636 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.521555    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b1bc322-d67e-468f-85df-750ba827986d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8bc5b857-9fh8b\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.521636 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.521573    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crchm\" (UniqueName: \"kubernetes.io/projected/0b1bc322-d67e-468f-85df-750ba827986d-kube-api-access-crchm\") pod \"isvc-canary-promote-predictor-8bc5b857-9fh8b\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.521636 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.521594    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a118f509-5afa-4920-9b48-42b7966034fb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-2gm85\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:42.521745 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.521666    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1bc322-d67e-468f-85df-750ba827986d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8bc5b857-9fh8b\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.521745 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.521693    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a118f509-5afa-4920-9b48-42b7966034fb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-2gm85\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:42.521745 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.521709    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b1bc322-d67e-468f-85df-750ba827986d-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-9fh8b\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.521745 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.521725    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25bwx\" (UniqueName: \"kubernetes.io/projected/a118f509-5afa-4920-9b48-42b7966034fb-kube-api-access-25bwx\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-2gm85\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:42.521871 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.521745    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b1bc322-d67e-468f-85df-750ba827986d-cabundle-cert\") pod \"isvc-canary-promote-predictor-8bc5b857-9fh8b\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.521871 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.521791    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a118f509-5afa-4920-9b48-42b7966034fb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-2gm85\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:42.583050 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.582992    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:03:42.622444 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.622424    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a118f509-5afa-4920-9b48-42b7966034fb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-2gm85\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:42.622546 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.622460    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1bc322-d67e-468f-85df-750ba827986d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8bc5b857-9fh8b\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.622546 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.622483    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a118f509-5afa-4920-9b48-42b7966034fb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-2gm85\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:42.622546 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.622506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b1bc322-d67e-468f-85df-750ba827986d-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-9fh8b\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.622733 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.622624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25bwx\" (UniqueName: \"kubernetes.io/projected/a118f509-5afa-4920-9b48-42b7966034fb-kube-api-access-25bwx\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-2gm85\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:42.622733 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.622665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b1bc322-d67e-468f-85df-750ba827986d-cabundle-cert\") pod \"isvc-canary-promote-predictor-8bc5b857-9fh8b\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.622733 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.622696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a118f509-5afa-4920-9b48-42b7966034fb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-2gm85\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:42.622894 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.622769    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a118f509-5afa-4920-9b48-42b7966034fb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-2gm85\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:42.622894 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.622801    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b1bc322-d67e-468f-85df-750ba827986d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8bc5b857-9fh8b\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.622894 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.622810    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a118f509-5afa-4920-9b48-42b7966034fb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-2gm85\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:42.622894 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.622829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crchm\" (UniqueName: \"kubernetes.io/projected/0b1bc322-d67e-468f-85df-750ba827986d-kube-api-access-crchm\") pod \"isvc-canary-promote-predictor-8bc5b857-9fh8b\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.622894 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.622860    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1bc322-d67e-468f-85df-750ba827986d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8bc5b857-9fh8b\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.623184 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:03:42.622900    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 17:03:42.623184 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:03:42.622983    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a118f509-5afa-4920-9b48-42b7966034fb-proxy-tls podName:a118f509-5afa-4920-9b48-42b7966034fb nodeName:}" failed. No retries permitted until 2026-09-17 17:03:43.122964512 +0000 UTC m=+1288.689076227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a118f509-5afa-4920-9b48-42b7966034fb-proxy-tls") pod "isvc-canary-promote-v2-predictor-65744c97b6-2gm85" (UID: "a118f509-5afa-4920-9b48-42b7966034fb") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 17:03:42.623299 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.623247    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a118f509-5afa-4920-9b48-42b7966034fb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-2gm85\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:42.623404 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.623382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b1bc322-d67e-468f-85df-750ba827986d-cabundle-cert\") pod \"isvc-canary-promote-predictor-8bc5b857-9fh8b\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.623482 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.623467    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a118f509-5afa-4920-9b48-42b7966034fb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-2gm85\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:42.623572 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.623549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b1bc322-d67e-468f-85df-750ba827986d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8bc5b857-9fh8b\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.625246 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.625226    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b1bc322-d67e-468f-85df-750ba827986d-proxy-tls\") pod \"isvc-canary-promote-predictor-8bc5b857-9fh8b\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.633663 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.633643    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25bwx\" (UniqueName: \"kubernetes.io/projected/a118f509-5afa-4920-9b48-42b7966034fb-kube-api-access-25bwx\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-2gm85\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:42.633771 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.633755    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crchm\" (UniqueName: \"kubernetes.io/projected/0b1bc322-d67e-468f-85df-750ba827986d-kube-api-access-crchm\") pod \"isvc-canary-promote-predictor-8bc5b857-9fh8b\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.767010 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.766987    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:03:42.809045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.809023    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:42.824645 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:42.824622    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57661324-5764-4d10-b56a-9d36852c19ac" path="/var/lib/kubelet/pods/57661324-5764-4d10-b56a-9d36852c19ac/volumes"
Sep 17 17:03:43.078498 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:43.078466    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"]
Sep 17 17:03:43.081555 ip-10-0-132-85 kubenswrapper[2565]: W0917 17:03:43.081530    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1bc322_d67e_468f_85df_750ba827986d.slice/crio-9b7eb89e8815182988bc9e3e782d2bdbef785a1924f9ae3e024023e1367a7a7e WatchSource:0}: Error finding container 9b7eb89e8815182988bc9e3e782d2bdbef785a1924f9ae3e024023e1367a7a7e: Status 404 returned error can't find the container with id 9b7eb89e8815182988bc9e3e782d2bdbef785a1924f9ae3e024023e1367a7a7e
Sep 17 17:03:43.127364 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:43.127342    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a118f509-5afa-4920-9b48-42b7966034fb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-2gm85\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:43.129415 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:43.129395    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a118f509-5afa-4920-9b48-42b7966034fb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-65744c97b6-2gm85\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:43.283279 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:43.283243    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" event={"ID":"0b1bc322-d67e-468f-85df-750ba827986d","Type":"ContainerStarted","Data":"535f3cd774d0f4c9d1d08e2f1e0f1a7aafc4383dbaf1a450208847ec0f029186"}
Sep 17 17:03:43.283614 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:43.283303    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" event={"ID":"0b1bc322-d67e-468f-85df-750ba827986d","Type":"ContainerStarted","Data":"9b7eb89e8815182988bc9e3e782d2bdbef785a1924f9ae3e024023e1367a7a7e"}
Sep 17 17:03:43.401701 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:43.401634    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:43.674874 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:43.674848    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"]
Sep 17 17:03:43.676976 ip-10-0-132-85 kubenswrapper[2565]: W0917 17:03:43.676944    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda118f509_5afa_4920_9b48_42b7966034fb.slice/crio-7cf8a415fa2da4bf1eb20acef17e989275a75c07ef3712ae60619d3a5b208b2d WatchSource:0}: Error finding container 7cf8a415fa2da4bf1eb20acef17e989275a75c07ef3712ae60619d3a5b208b2d: Status 404 returned error can't find the container with id 7cf8a415fa2da4bf1eb20acef17e989275a75c07ef3712ae60619d3a5b208b2d
Sep 17 17:03:44.289552 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:44.289507    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" event={"ID":"a118f509-5afa-4920-9b48-42b7966034fb","Type":"ContainerStarted","Data":"21dcfd56b4152df91d8ea4dcb50300f1b522756a9b5fcbdbf08ecfbb258fae8a"}
Sep 17 17:03:44.289905 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:44.289593    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" event={"ID":"a118f509-5afa-4920-9b48-42b7966034fb","Type":"ContainerStarted","Data":"7cf8a415fa2da4bf1eb20acef17e989275a75c07ef3712ae60619d3a5b208b2d"}
Sep 17 17:03:47.304077 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:47.304035    2565 generic.go:356] "Generic (PLEG): container finished" podID="0b1bc322-d67e-468f-85df-750ba827986d" containerID="535f3cd774d0f4c9d1d08e2f1e0f1a7aafc4383dbaf1a450208847ec0f029186" exitCode=0
Sep 17 17:03:47.304441 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:47.304080    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" event={"ID":"0b1bc322-d67e-468f-85df-750ba827986d","Type":"ContainerDied","Data":"535f3cd774d0f4c9d1d08e2f1e0f1a7aafc4383dbaf1a450208847ec0f029186"}
Sep 17 17:03:48.309853 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:48.309822    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" event={"ID":"0b1bc322-d67e-468f-85df-750ba827986d","Type":"ContainerStarted","Data":"513981fc0be676f1d9a735dfceff00629f8cba90f4d873e670a671b8e9fb9693"}
Sep 17 17:03:48.310235 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:48.309882    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" event={"ID":"0b1bc322-d67e-468f-85df-750ba827986d","Type":"ContainerStarted","Data":"c4e95f76cf22513c595d21915a9a1ff0e64d31fe3ea5cf702eb687ec9d2ac5e6"}
Sep 17 17:03:48.310235 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:48.310114    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:48.311364 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:48.311344    2565 generic.go:356] "Generic (PLEG): container finished" podID="a118f509-5afa-4920-9b48-42b7966034fb" containerID="21dcfd56b4152df91d8ea4dcb50300f1b522756a9b5fcbdbf08ecfbb258fae8a" exitCode=0
Sep 17 17:03:48.311441 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:48.311415    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" event={"ID":"a118f509-5afa-4920-9b48-42b7966034fb","Type":"ContainerDied","Data":"21dcfd56b4152df91d8ea4dcb50300f1b522756a9b5fcbdbf08ecfbb258fae8a"}
Sep 17 17:03:48.328042 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:48.327992    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" podStartSLOduration=2.629731768 podStartE2EDuration="6.327974006s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.698242238s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:03:48.327778155 +0000 UTC m=+1293.893889877" watchObservedRunningTime="2026-09-17 17:03:48.327974006 +0000 UTC m=+1293.894085725"
Sep 17 17:03:49.317479 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:49.317439    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" event={"ID":"a118f509-5afa-4920-9b48-42b7966034fb","Type":"ContainerStarted","Data":"7ed37d41218776a93da4f75cb9e33dcf43cdc5d08807f0b508e9f50e767209e3"}
Sep 17 17:03:49.317971 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:49.317499    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" event={"ID":"a118f509-5afa-4920-9b48-42b7966034fb","Type":"ContainerStarted","Data":"92c4ecb1e5d6b7b098b688de48503835f65c24c5ae009b594299cf839caf136b"}
Sep 17 17:03:49.317971 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:49.317703    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:49.318093 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:49.318017    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:49.318199 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:49.318178    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:49.318958 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:49.318930    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 17:03:49.319195 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:49.319172    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:03:49.335016 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:49.334970    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" podStartSLOduration=3.740739258 podStartE2EDuration="7.334953398s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.59421414s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:03:49.334524563 +0000 UTC m=+1294.900636289" watchObservedRunningTime="2026-09-17 17:03:49.334953398 +0000 UTC m=+1294.901065122"
Sep 17 17:03:50.321897 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:50.321855    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:03:50.322331 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:50.322138    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 17:03:51.326443 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:51.326395    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:03:55.327527 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:55.327497    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:03:55.328162 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:55.328129    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 17:03:56.332712 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:56.332676    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:03:56.334102 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:03:56.334070    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:04:05.329017 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:04:05.328972    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 17:04:06.334761 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:04:06.334719    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:04:15.328165 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:04:15.328117    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 17:04:16.334113 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:04:16.334071    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:04:25.328401 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:04:25.328353    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 17:04:26.334144 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:04:26.334102    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:04:35.328377 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:04:35.328293    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 17:04:36.334479 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:04:36.334441    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:04:45.328637 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:04:45.328598    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 17:04:46.334198 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:04:46.334160    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:04:55.328635 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:04:55.328595    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 17:04:56.333961 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:04:56.333899    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:05:01.453826 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:01.453788    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:05:01.819419 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:01.819338    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:05:06.334116 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:06.334082    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:05:12.558147 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:12.558114    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"]
Sep 17 17:05:12.558552 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:12.558418    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kserve-container" containerID="cri-o://c4e95f76cf22513c595d21915a9a1ff0e64d31fe3ea5cf702eb687ec9d2ac5e6" gracePeriod=30
Sep 17 17:05:12.558552 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:12.558459    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kube-rbac-proxy" containerID="cri-o://513981fc0be676f1d9a735dfceff00629f8cba90f4d873e670a671b8e9fb9693" gracePeriod=30
Sep 17 17:05:12.577706 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:12.577684    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:05:13.653379 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:13.653345    2565 generic.go:356] "Generic (PLEG): container finished" podID="0b1bc322-d67e-468f-85df-750ba827986d" containerID="513981fc0be676f1d9a735dfceff00629f8cba90f4d873e670a671b8e9fb9693" exitCode=2
Sep 17 17:05:13.653725 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:13.653392    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" event={"ID":"0b1bc322-d67e-468f-85df-750ba827986d","Type":"ContainerDied","Data":"513981fc0be676f1d9a735dfceff00629f8cba90f4d873e670a671b8e9fb9693"}
Sep 17 17:05:15.322166 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:15.322114    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 17:05:19.684453 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.684414    2565 generic.go:356] "Generic (PLEG): container finished" podID="0b1bc322-d67e-468f-85df-750ba827986d" containerID="c4e95f76cf22513c595d21915a9a1ff0e64d31fe3ea5cf702eb687ec9d2ac5e6" exitCode=0
Sep 17 17:05:19.684756 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.684465    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" event={"ID":"0b1bc322-d67e-468f-85df-750ba827986d","Type":"ContainerDied","Data":"c4e95f76cf22513c595d21915a9a1ff0e64d31fe3ea5cf702eb687ec9d2ac5e6"}
Sep 17 17:05:19.794022 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.794001    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:05:19.828534 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.828508    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b1bc322-d67e-468f-85df-750ba827986d-proxy-tls\") pod \"0b1bc322-d67e-468f-85df-750ba827986d\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") "
Sep 17 17:05:19.828679 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.828549    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crchm\" (UniqueName: \"kubernetes.io/projected/0b1bc322-d67e-468f-85df-750ba827986d-kube-api-access-crchm\") pod \"0b1bc322-d67e-468f-85df-750ba827986d\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") "
Sep 17 17:05:19.828679 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.828592    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b1bc322-d67e-468f-85df-750ba827986d-kube-rbac-proxy-sar-config\") pod \"0b1bc322-d67e-468f-85df-750ba827986d\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") "
Sep 17 17:05:19.828813 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.828752    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1bc322-d67e-468f-85df-750ba827986d-kserve-provision-location\") pod \"0b1bc322-d67e-468f-85df-750ba827986d\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") "
Sep 17 17:05:19.828813 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.828800    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b1bc322-d67e-468f-85df-750ba827986d-cabundle-cert\") pod \"0b1bc322-d67e-468f-85df-750ba827986d\" (UID: \"0b1bc322-d67e-468f-85df-750ba827986d\") "
Sep 17 17:05:19.828942 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.828882    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1bc322-d67e-468f-85df-750ba827986d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b1bc322-d67e-468f-85df-750ba827986d" (UID: "0b1bc322-d67e-468f-85df-750ba827986d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:05:19.829124 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.829101    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b1bc322-d67e-468f-85df-750ba827986d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:05:19.829236 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.829170    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1bc322-d67e-468f-85df-750ba827986d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b1bc322-d67e-468f-85df-750ba827986d" (UID: "0b1bc322-d67e-468f-85df-750ba827986d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:05:19.829327 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.829307    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1bc322-d67e-468f-85df-750ba827986d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b1bc322-d67e-468f-85df-750ba827986d" (UID: "0b1bc322-d67e-468f-85df-750ba827986d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:05:19.830698 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.830666    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1bc322-d67e-468f-85df-750ba827986d-kube-api-access-crchm" (OuterVolumeSpecName: "kube-api-access-crchm") pod "0b1bc322-d67e-468f-85df-750ba827986d" (UID: "0b1bc322-d67e-468f-85df-750ba827986d"). InnerVolumeSpecName "kube-api-access-crchm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:05:19.830954 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.830933    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1bc322-d67e-468f-85df-750ba827986d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b1bc322-d67e-468f-85df-750ba827986d" (UID: "0b1bc322-d67e-468f-85df-750ba827986d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:05:19.930224 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.930178    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b1bc322-d67e-468f-85df-750ba827986d-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:05:19.930224 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.930197    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crchm\" (UniqueName: \"kubernetes.io/projected/0b1bc322-d67e-468f-85df-750ba827986d-kube-api-access-crchm\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:05:19.930224 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.930208    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1bc322-d67e-468f-85df-750ba827986d-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:05:19.930367 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:19.930296    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b1bc322-d67e-468f-85df-750ba827986d-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:05:20.690020 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:20.689985    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b" event={"ID":"0b1bc322-d67e-468f-85df-750ba827986d","Type":"ContainerDied","Data":"9b7eb89e8815182988bc9e3e782d2bdbef785a1924f9ae3e024023e1367a7a7e"}
Sep 17 17:05:20.690401 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:20.690038    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"
Sep 17 17:05:20.690401 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:20.690062    2565 scope.go:117] "RemoveContainer" containerID="513981fc0be676f1d9a735dfceff00629f8cba90f4d873e670a671b8e9fb9693"
Sep 17 17:05:20.698962 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:20.698938    2565 scope.go:117] "RemoveContainer" containerID="c4e95f76cf22513c595d21915a9a1ff0e64d31fe3ea5cf702eb687ec9d2ac5e6"
Sep 17 17:05:20.706365 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:20.706348    2565 scope.go:117] "RemoveContainer" containerID="535f3cd774d0f4c9d1d08e2f1e0f1a7aafc4383dbaf1a450208847ec0f029186"
Sep 17 17:05:20.711467 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:20.711446    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"]
Sep 17 17:05:20.715750 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:20.715728    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8bc5b857-9fh8b"]
Sep 17 17:05:20.823190 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:20.823168    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b1bc322-d67e-468f-85df-750ba827986d" path="/var/lib/kubelet/pods/0b1bc322-d67e-468f-85df-750ba827986d/volumes"
Sep 17 17:05:22.135228 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:05:22.135197    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:06:42.767004 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:42.766969    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"]
Sep 17 17:06:42.767606 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:42.767567    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kserve-container" containerID="cri-o://92c4ecb1e5d6b7b098b688de48503835f65c24c5ae009b594299cf839caf136b" gracePeriod=30
Sep 17 17:06:42.768308 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:42.767866    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kube-rbac-proxy" containerID="cri-o://7ed37d41218776a93da4f75cb9e33dcf43cdc5d08807f0b508e9f50e767209e3" gracePeriod=30
Sep 17 17:06:43.013303 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:43.013277    2565 generic.go:356] "Generic (PLEG): container finished" podID="a118f509-5afa-4920-9b48-42b7966034fb" containerID="7ed37d41218776a93da4f75cb9e33dcf43cdc5d08807f0b508e9f50e767209e3" exitCode=2
Sep 17 17:06:43.013398 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:43.013330    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" event={"ID":"a118f509-5afa-4920-9b48-42b7966034fb","Type":"ContainerDied","Data":"7ed37d41218776a93da4f75cb9e33dcf43cdc5d08807f0b508e9f50e767209e3"}
Sep 17 17:06:43.556626 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:43.556594    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:06:46.327117 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.327076    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 17:06:46.334102 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.334073    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 17:06:46.513162 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.513139    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:06:46.538701 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.538678    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a118f509-5afa-4920-9b48-42b7966034fb-cabundle-cert\") pod \"a118f509-5afa-4920-9b48-42b7966034fb\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") "
Sep 17 17:06:46.538833 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.538709    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a118f509-5afa-4920-9b48-42b7966034fb-proxy-tls\") pod \"a118f509-5afa-4920-9b48-42b7966034fb\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") "
Sep 17 17:06:46.538833 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.538738    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a118f509-5afa-4920-9b48-42b7966034fb-kserve-provision-location\") pod \"a118f509-5afa-4920-9b48-42b7966034fb\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") "
Sep 17 17:06:46.538833 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.538784    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25bwx\" (UniqueName: \"kubernetes.io/projected/a118f509-5afa-4920-9b48-42b7966034fb-kube-api-access-25bwx\") pod \"a118f509-5afa-4920-9b48-42b7966034fb\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") "
Sep 17 17:06:46.538833 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.538801    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a118f509-5afa-4920-9b48-42b7966034fb-kube-rbac-proxy-sar-config\") pod \"a118f509-5afa-4920-9b48-42b7966034fb\" (UID: \"a118f509-5afa-4920-9b48-42b7966034fb\") "
Sep 17 17:06:46.539178 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.539147    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a118f509-5afa-4920-9b48-42b7966034fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a118f509-5afa-4920-9b48-42b7966034fb" (UID: "a118f509-5afa-4920-9b48-42b7966034fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:06:46.539262 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.539149    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a118f509-5afa-4920-9b48-42b7966034fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a118f509-5afa-4920-9b48-42b7966034fb" (UID: "a118f509-5afa-4920-9b48-42b7966034fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:06:46.539262 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.539228    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a118f509-5afa-4920-9b48-42b7966034fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a118f509-5afa-4920-9b48-42b7966034fb" (UID: "a118f509-5afa-4920-9b48-42b7966034fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:06:46.540685 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.540658    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a118f509-5afa-4920-9b48-42b7966034fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a118f509-5afa-4920-9b48-42b7966034fb" (UID: "a118f509-5afa-4920-9b48-42b7966034fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:06:46.540829 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.540807    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a118f509-5afa-4920-9b48-42b7966034fb-kube-api-access-25bwx" (OuterVolumeSpecName: "kube-api-access-25bwx") pod "a118f509-5afa-4920-9b48-42b7966034fb" (UID: "a118f509-5afa-4920-9b48-42b7966034fb"). InnerVolumeSpecName "kube-api-access-25bwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:06:46.640112 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.640086    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a118f509-5afa-4920-9b48-42b7966034fb-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:06:46.640222 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.640115    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25bwx\" (UniqueName: \"kubernetes.io/projected/a118f509-5afa-4920-9b48-42b7966034fb-kube-api-access-25bwx\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:06:46.640222 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.640130    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a118f509-5afa-4920-9b48-42b7966034fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:06:46.640222 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.640146    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a118f509-5afa-4920-9b48-42b7966034fb-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:06:46.640222 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:46.640159    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a118f509-5afa-4920-9b48-42b7966034fb-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:06:47.028595 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:47.028534    2565 generic.go:356] "Generic (PLEG): container finished" podID="a118f509-5afa-4920-9b48-42b7966034fb" containerID="92c4ecb1e5d6b7b098b688de48503835f65c24c5ae009b594299cf839caf136b" exitCode=0
Sep 17 17:06:47.028595 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:47.028586    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"
Sep 17 17:06:47.028732 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:47.028617    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" event={"ID":"a118f509-5afa-4920-9b48-42b7966034fb","Type":"ContainerDied","Data":"92c4ecb1e5d6b7b098b688de48503835f65c24c5ae009b594299cf839caf136b"}
Sep 17 17:06:47.028732 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:47.028672    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85" event={"ID":"a118f509-5afa-4920-9b48-42b7966034fb","Type":"ContainerDied","Data":"7cf8a415fa2da4bf1eb20acef17e989275a75c07ef3712ae60619d3a5b208b2d"}
Sep 17 17:06:47.028732 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:47.028699    2565 scope.go:117] "RemoveContainer" containerID="7ed37d41218776a93da4f75cb9e33dcf43cdc5d08807f0b508e9f50e767209e3"
Sep 17 17:06:47.036268 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:47.036222    2565 scope.go:117] "RemoveContainer" containerID="92c4ecb1e5d6b7b098b688de48503835f65c24c5ae009b594299cf839caf136b"
Sep 17 17:06:47.043234 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:47.043214    2565 scope.go:117] "RemoveContainer" containerID="21dcfd56b4152df91d8ea4dcb50300f1b522756a9b5fcbdbf08ecfbb258fae8a"
Sep 17 17:06:47.048182 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:47.048155    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"]
Sep 17 17:06:47.050848 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:47.050832    2565 scope.go:117] "RemoveContainer" containerID="7ed37d41218776a93da4f75cb9e33dcf43cdc5d08807f0b508e9f50e767209e3"
Sep 17 17:06:47.051141 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:06:47.051116    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed37d41218776a93da4f75cb9e33dcf43cdc5d08807f0b508e9f50e767209e3\": container with ID starting with 7ed37d41218776a93da4f75cb9e33dcf43cdc5d08807f0b508e9f50e767209e3 not found: ID does not exist" containerID="7ed37d41218776a93da4f75cb9e33dcf43cdc5d08807f0b508e9f50e767209e3"
Sep 17 17:06:47.051275 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:47.051147    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed37d41218776a93da4f75cb9e33dcf43cdc5d08807f0b508e9f50e767209e3"} err="failed to get container status \"7ed37d41218776a93da4f75cb9e33dcf43cdc5d08807f0b508e9f50e767209e3\": rpc error: code = NotFound desc = could not find container \"7ed37d41218776a93da4f75cb9e33dcf43cdc5d08807f0b508e9f50e767209e3\": container with ID starting with 7ed37d41218776a93da4f75cb9e33dcf43cdc5d08807f0b508e9f50e767209e3 not found: ID does not exist"
Sep 17 17:06:47.051275 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:47.051165    2565 scope.go:117] "RemoveContainer" containerID="92c4ecb1e5d6b7b098b688de48503835f65c24c5ae009b594299cf839caf136b"
Sep 17 17:06:47.051427 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:06:47.051412    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c4ecb1e5d6b7b098b688de48503835f65c24c5ae009b594299cf839caf136b\": container with ID starting with 92c4ecb1e5d6b7b098b688de48503835f65c24c5ae009b594299cf839caf136b not found: ID does not exist" containerID="92c4ecb1e5d6b7b098b688de48503835f65c24c5ae009b594299cf839caf136b"
Sep 17 17:06:47.051509 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:47.051429    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c4ecb1e5d6b7b098b688de48503835f65c24c5ae009b594299cf839caf136b"} err="failed to get container status \"92c4ecb1e5d6b7b098b688de48503835f65c24c5ae009b594299cf839caf136b\": rpc error: code = NotFound desc = could not find container \"92c4ecb1e5d6b7b098b688de48503835f65c24c5ae009b594299cf839caf136b\": container with ID starting with 92c4ecb1e5d6b7b098b688de48503835f65c24c5ae009b594299cf839caf136b not found: ID does not exist"
Sep 17 17:06:47.051509 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:47.051441    2565 scope.go:117] "RemoveContainer" containerID="21dcfd56b4152df91d8ea4dcb50300f1b522756a9b5fcbdbf08ecfbb258fae8a"
Sep 17 17:06:47.051664 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:06:47.051645    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21dcfd56b4152df91d8ea4dcb50300f1b522756a9b5fcbdbf08ecfbb258fae8a\": container with ID starting with 21dcfd56b4152df91d8ea4dcb50300f1b522756a9b5fcbdbf08ecfbb258fae8a not found: ID does not exist" containerID="21dcfd56b4152df91d8ea4dcb50300f1b522756a9b5fcbdbf08ecfbb258fae8a"
Sep 17 17:06:47.051744 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:47.051667    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21dcfd56b4152df91d8ea4dcb50300f1b522756a9b5fcbdbf08ecfbb258fae8a"} err="failed to get container status \"21dcfd56b4152df91d8ea4dcb50300f1b522756a9b5fcbdbf08ecfbb258fae8a\": rpc error: code = NotFound desc = could not find container \"21dcfd56b4152df91d8ea4dcb50300f1b522756a9b5fcbdbf08ecfbb258fae8a\": container with ID starting with 21dcfd56b4152df91d8ea4dcb50300f1b522756a9b5fcbdbf08ecfbb258fae8a not found: ID does not exist"
Sep 17 17:06:47.052179 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:47.052159    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-65744c97b6-2gm85"]
Sep 17 17:06:48.823280 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:48.823242    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a118f509-5afa-4920-9b48-42b7966034fb" path="/var/lib/kubelet/pods/a118f509-5afa-4920-9b48-42b7966034fb/volumes"
Sep 17 17:06:56.959694 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.959662    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:06:56.994967 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.994937    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"]
Sep 17 17:06:56.995234 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.995223    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="storage-initializer"
Sep 17 17:06:56.995281 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.995236    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="storage-initializer"
Sep 17 17:06:56.995281 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.995245    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kube-rbac-proxy"
Sep 17 17:06:56.995281 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.995251    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kube-rbac-proxy"
Sep 17 17:06:56.995281 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.995259    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kube-rbac-proxy"
Sep 17 17:06:56.995281 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.995265    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kube-rbac-proxy"
Sep 17 17:06:56.995281 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.995278    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="storage-initializer"
Sep 17 17:06:56.995498 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.995284    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="storage-initializer"
Sep 17 17:06:56.995498 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.995290    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kserve-container"
Sep 17 17:06:56.995498 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.995295    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kserve-container"
Sep 17 17:06:56.995498 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.995306    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kserve-container"
Sep 17 17:06:56.995498 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.995311    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kserve-container"
Sep 17 17:06:56.995498 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.995354    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kserve-container"
Sep 17 17:06:56.995498 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.995363    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kserve-container"
Sep 17 17:06:56.995498 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.995368    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a118f509-5afa-4920-9b48-42b7966034fb" containerName="kube-rbac-proxy"
Sep 17 17:06:56.995498 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.995375    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b1bc322-d67e-468f-85df-750ba827986d" containerName="kube-rbac-proxy"
Sep 17 17:06:56.999658 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:56.999640    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:06:57.002115 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.002091    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 17 17:06:57.002115 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.002105    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:06:57.002280 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.002091    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:06:57.003045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.003026    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 17:06:57.003045 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.003033    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlgx8\""
Sep 17 17:06:57.006585 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.006563    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"]
Sep 17 17:06:57.012713 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.012694    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k45l8\" (UniqueName: \"kubernetes.io/projected/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-kube-api-access-k45l8\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-8dskw\" (UID: \"fd9ec3df-30f4-47e5-95fb-e2647d88bef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:06:57.012817 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.012740    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-8dskw\" (UID: \"fd9ec3df-30f4-47e5-95fb-e2647d88bef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:06:57.012817 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.012761    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-8dskw\" (UID: \"fd9ec3df-30f4-47e5-95fb-e2647d88bef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:06:57.113646 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.113626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-8dskw\" (UID: \"fd9ec3df-30f4-47e5-95fb-e2647d88bef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:06:57.113768 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.113654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-8dskw\" (UID: \"fd9ec3df-30f4-47e5-95fb-e2647d88bef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:06:57.113768 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.113691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k45l8\" (UniqueName: \"kubernetes.io/projected/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-kube-api-access-k45l8\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-8dskw\" (UID: \"fd9ec3df-30f4-47e5-95fb-e2647d88bef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:06:57.113870 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:06:57.113765    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-serving-cert: secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Sep 17 17:06:57.113870 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:06:57.113831    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-proxy-tls podName:fd9ec3df-30f4-47e5-95fb-e2647d88bef2 nodeName:}" failed. No retries permitted until 2026-09-17 17:06:57.613813752 +0000 UTC m=+1483.179925458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-proxy-tls") pod "message-dumper-raw-cipn-predictor-9774bf55d-8dskw" (UID: "fd9ec3df-30f4-47e5-95fb-e2647d88bef2") : secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Sep 17 17:06:57.114234 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.114218    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-8dskw\" (UID: \"fd9ec3df-30f4-47e5-95fb-e2647d88bef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:06:57.122050 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.122033    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k45l8\" (UniqueName: \"kubernetes.io/projected/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-kube-api-access-k45l8\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-8dskw\" (UID: \"fd9ec3df-30f4-47e5-95fb-e2647d88bef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:06:57.349479 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.349423    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:06:57.618082 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.618062    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-8dskw\" (UID: \"fd9ec3df-30f4-47e5-95fb-e2647d88bef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:06:57.620367 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.620348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-8dskw\" (UID: \"fd9ec3df-30f4-47e5-95fb-e2647d88bef2\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:06:57.909822 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:57.909766    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:06:58.177618 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:58.177588    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"]
Sep 17 17:06:58.180160 ip-10-0-132-85 kubenswrapper[2565]: W0917 17:06:58.180126    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9ec3df_30f4_47e5_95fb_e2647d88bef2.slice/crio-58be0736bb62eddb895a8648f7a05de8d5c5c7847a314b89b5e413dc64b850c4 WatchSource:0}: Error finding container 58be0736bb62eddb895a8648f7a05de8d5c5c7847a314b89b5e413dc64b850c4: Status 404 returned error can't find the container with id 58be0736bb62eddb895a8648f7a05de8d5c5c7847a314b89b5e413dc64b850c4
Sep 17 17:06:59.077251 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:59.077204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw" event={"ID":"fd9ec3df-30f4-47e5-95fb-e2647d88bef2","Type":"ContainerStarted","Data":"1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2"}
Sep 17 17:06:59.077416 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:59.077286    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw" event={"ID":"fd9ec3df-30f4-47e5-95fb-e2647d88bef2","Type":"ContainerStarted","Data":"740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952"}
Sep 17 17:06:59.077416 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:59.077319    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw" event={"ID":"fd9ec3df-30f4-47e5-95fb-e2647d88bef2","Type":"ContainerStarted","Data":"58be0736bb62eddb895a8648f7a05de8d5c5c7847a314b89b5e413dc64b850c4"}
Sep 17 17:06:59.077416 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:59.077353    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:06:59.096775 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:06:59.096709    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw" podStartSLOduration=3.096692933 podStartE2EDuration="3.096692933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:06:59.095409721 +0000 UTC m=+1484.661521447" watchObservedRunningTime="2026-09-17 17:06:59.096692933 +0000 UTC m=+1484.662804652"
Sep 17 17:07:00.081479 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:00.081444    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:07:00.083382 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:00.083357    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:07:07.095202 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:07.095175    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:07:16.994579 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:16.994551    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:07:17.036387 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.036356    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"]
Sep 17 17:07:17.041142 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.041127    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.043817 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.043793    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 17:07:17.044033 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.044016    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:07:17.044113 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.044057    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 17 17:07:17.044113 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.044016    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:07:17.052031 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.052007    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"]
Sep 17 17:07:17.152507 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.152476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c96fbed-a557-4494-b2f6-e79754072b6b-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.152636 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.152556    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c96fbed-a557-4494-b2f6-e79754072b6b-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.152636 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.152612    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c96fbed-a557-4494-b2f6-e79754072b6b-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.152734 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.152654    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kthn\" (UniqueName: \"kubernetes.io/projected/5c96fbed-a557-4494-b2f6-e79754072b6b-kube-api-access-4kthn\") pod \"isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.152734 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.152696    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c96fbed-a557-4494-b2f6-e79754072b6b-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.221258 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.221193    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:07:17.253162 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.253143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c96fbed-a557-4494-b2f6-e79754072b6b-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.253279 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.253177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c96fbed-a557-4494-b2f6-e79754072b6b-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.253279 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.253199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kthn\" (UniqueName: \"kubernetes.io/projected/5c96fbed-a557-4494-b2f6-e79754072b6b-kube-api-access-4kthn\") pod \"isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.253279 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.253229    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c96fbed-a557-4494-b2f6-e79754072b6b-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.253279 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.253264    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c96fbed-a557-4494-b2f6-e79754072b6b-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.253613 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.253592    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c96fbed-a557-4494-b2f6-e79754072b6b-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.253908 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.253884    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c96fbed-a557-4494-b2f6-e79754072b6b-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.254024 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.253972    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c96fbed-a557-4494-b2f6-e79754072b6b-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.255667 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.255650    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c96fbed-a557-4494-b2f6-e79754072b6b-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.261701 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.261683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kthn\" (UniqueName: \"kubernetes.io/projected/5c96fbed-a557-4494-b2f6-e79754072b6b-kube-api-access-4kthn\") pod \"isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.350926 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.350896    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:17.621942 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:17.621898    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"]
Sep 17 17:07:17.624167 ip-10-0-132-85 kubenswrapper[2565]: W0917 17:07:17.624131    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c96fbed_a557_4494_b2f6_e79754072b6b.slice/crio-d897366c0c23025fcf63662c0969c8ce2b7110c932fe9da10bb2006f816600eb WatchSource:0}: Error finding container d897366c0c23025fcf63662c0969c8ce2b7110c932fe9da10bb2006f816600eb: Status 404 returned error can't find the container with id d897366c0c23025fcf63662c0969c8ce2b7110c932fe9da10bb2006f816600eb
Sep 17 17:07:18.157978 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:18.157946    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" event={"ID":"5c96fbed-a557-4494-b2f6-e79754072b6b","Type":"ContainerStarted","Data":"d3f25d1c40e71c60d9ad14a25523cac7731532afe511239240db2c7b7d2b8b7b"}
Sep 17 17:07:18.158354 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:18.158005    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" event={"ID":"5c96fbed-a557-4494-b2f6-e79754072b6b","Type":"ContainerStarted","Data":"d897366c0c23025fcf63662c0969c8ce2b7110c932fe9da10bb2006f816600eb"}
Sep 17 17:07:22.174472 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:22.174440    2565 generic.go:356] "Generic (PLEG): container finished" podID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerID="d3f25d1c40e71c60d9ad14a25523cac7731532afe511239240db2c7b7d2b8b7b" exitCode=0
Sep 17 17:07:22.174895 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:22.174490    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" event={"ID":"5c96fbed-a557-4494-b2f6-e79754072b6b","Type":"ContainerDied","Data":"d3f25d1c40e71c60d9ad14a25523cac7731532afe511239240db2c7b7d2b8b7b"}
Sep 17 17:07:23.180168 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:23.180135    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" event={"ID":"5c96fbed-a557-4494-b2f6-e79754072b6b","Type":"ContainerStarted","Data":"51e090358000e1d605b5bb245e439202c60cf9d8c646ecf3aa53cfe862c6e3a9"}
Sep 17 17:07:23.180616 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:23.180242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" event={"ID":"5c96fbed-a557-4494-b2f6-e79754072b6b","Type":"ContainerStarted","Data":"87d348b7be36aa37e6500124dc61d2d6f45231d138b1d0b9b6512a98df337528"}
Sep 17 17:07:23.180616 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:23.180286    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" event={"ID":"5c96fbed-a557-4494-b2f6-e79754072b6b","Type":"ContainerStarted","Data":"e88a3f60c1db5b8026601e2e4d4c6fead5c1e6a9bc05001ede54e4987c2066d9"}
Sep 17 17:07:23.180616 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:23.180519    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:23.201716 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:23.201674    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podStartSLOduration=2.21288308 podStartE2EDuration="6.201660193s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.988777113s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:07:23.199641471 +0000 UTC m=+1508.765753202" watchObservedRunningTime="2026-09-17 17:07:23.201660193 +0000 UTC m=+1508.767771967"
Sep 17 17:07:24.184318 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:24.184275    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:24.184753 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:24.184350    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:24.185576 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:24.185536    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 17:07:24.186175 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:24.186149    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:07:25.188121 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:25.188075    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 17:07:25.188502 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:25.188381    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:07:30.191960 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:30.191933    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:07:30.194356 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:30.192529    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 17:07:30.194356 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:30.192830    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:07:40.192484 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:40.192435    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 17:07:40.192883 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:40.192822    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:07:50.192798 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:50.192758    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 17:07:50.193306 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:07:50.193179    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:08:00.192458 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:00.192414    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 17:08:00.192856 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:00.192811    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:08:10.192465 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:10.192414    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 17:08:10.192881 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:10.192837    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:08:20.192804 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:20.192761    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 17:08:20.193292 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:20.193230    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:08:30.193330 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:30.193285    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 17:08:30.193838 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:30.193813    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:08:40.193534 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:40.193504    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:08:40.194006 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:40.193983    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:08:52.059510 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.059473    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-8dskw_fd9ec3df-30f4-47e5-95fb-e2647d88bef2/kserve-container/0.log"
Sep 17 17:08:52.242452 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.242420    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"]
Sep 17 17:08:52.242691 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.242656    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kserve-container" containerID="cri-o://e88a3f60c1db5b8026601e2e4d4c6fead5c1e6a9bc05001ede54e4987c2066d9" gracePeriod=30
Sep 17 17:08:52.242824 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.242694    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="agent" containerID="cri-o://51e090358000e1d605b5bb245e439202c60cf9d8c646ecf3aa53cfe862c6e3a9" gracePeriod=30
Sep 17 17:08:52.242824 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.242702    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kube-rbac-proxy" containerID="cri-o://87d348b7be36aa37e6500124dc61d2d6f45231d138b1d0b9b6512a98df337528" gracePeriod=30
Sep 17 17:08:52.263789 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.263766    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"]
Sep 17 17:08:52.264002 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.263984    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw" podUID="fd9ec3df-30f4-47e5-95fb-e2647d88bef2" containerName="kserve-container" containerID="cri-o://740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952" gracePeriod=30
Sep 17 17:08:52.264096 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.263992    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw" podUID="fd9ec3df-30f4-47e5-95fb-e2647d88bef2" containerName="kube-rbac-proxy" containerID="cri-o://1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2" gracePeriod=30
Sep 17 17:08:52.285559 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.285535    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:08:52.494422 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.494397    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:08:52.500071 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.500053    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-kube-rbac-proxy-sar-config\") pod \"fd9ec3df-30f4-47e5-95fb-e2647d88bef2\" (UID: \"fd9ec3df-30f4-47e5-95fb-e2647d88bef2\") "
Sep 17 17:08:52.500163 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.500099    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k45l8\" (UniqueName: \"kubernetes.io/projected/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-kube-api-access-k45l8\") pod \"fd9ec3df-30f4-47e5-95fb-e2647d88bef2\" (UID: \"fd9ec3df-30f4-47e5-95fb-e2647d88bef2\") "
Sep 17 17:08:52.500234 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.500216    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-proxy-tls\") pod \"fd9ec3df-30f4-47e5-95fb-e2647d88bef2\" (UID: \"fd9ec3df-30f4-47e5-95fb-e2647d88bef2\") "
Sep 17 17:08:52.500435 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.500412    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fd9ec3df-30f4-47e5-95fb-e2647d88bef2" (UID: "fd9ec3df-30f4-47e5-95fb-e2647d88bef2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:08:52.501992 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.501971    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-kube-api-access-k45l8" (OuterVolumeSpecName: "kube-api-access-k45l8") pod "fd9ec3df-30f4-47e5-95fb-e2647d88bef2" (UID: "fd9ec3df-30f4-47e5-95fb-e2647d88bef2"). InnerVolumeSpecName "kube-api-access-k45l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:08:52.502049 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.502035    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fd9ec3df-30f4-47e5-95fb-e2647d88bef2" (UID: "fd9ec3df-30f4-47e5-95fb-e2647d88bef2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:08:52.527255 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.527230    2565 generic.go:356] "Generic (PLEG): container finished" podID="fd9ec3df-30f4-47e5-95fb-e2647d88bef2" containerID="1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2" exitCode=2
Sep 17 17:08:52.527255 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.527250    2565 generic.go:356] "Generic (PLEG): container finished" podID="fd9ec3df-30f4-47e5-95fb-e2647d88bef2" containerID="740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952" exitCode=2
Sep 17 17:08:52.527434 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.527274    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"
Sep 17 17:08:52.527434 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.527304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw" event={"ID":"fd9ec3df-30f4-47e5-95fb-e2647d88bef2","Type":"ContainerDied","Data":"1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2"}
Sep 17 17:08:52.527434 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.527354    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw" event={"ID":"fd9ec3df-30f4-47e5-95fb-e2647d88bef2","Type":"ContainerDied","Data":"740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952"}
Sep 17 17:08:52.527434 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.527374    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw" event={"ID":"fd9ec3df-30f4-47e5-95fb-e2647d88bef2","Type":"ContainerDied","Data":"58be0736bb62eddb895a8648f7a05de8d5c5c7847a314b89b5e413dc64b850c4"}
Sep 17 17:08:52.527434 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.527399    2565 scope.go:117] "RemoveContainer" containerID="1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2"
Sep 17 17:08:52.530394 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.530360    2565 generic.go:356] "Generic (PLEG): container finished" podID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerID="87d348b7be36aa37e6500124dc61d2d6f45231d138b1d0b9b6512a98df337528" exitCode=2
Sep 17 17:08:52.530491 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.530395    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" event={"ID":"5c96fbed-a557-4494-b2f6-e79754072b6b","Type":"ContainerDied","Data":"87d348b7be36aa37e6500124dc61d2d6f45231d138b1d0b9b6512a98df337528"}
Sep 17 17:08:52.536345 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.536326    2565 scope.go:117] "RemoveContainer" containerID="740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952"
Sep 17 17:08:52.543247 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.543230    2565 scope.go:117] "RemoveContainer" containerID="1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2"
Sep 17 17:08:52.543487 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:08:52.543467    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2\": container with ID starting with 1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2 not found: ID does not exist" containerID="1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2"
Sep 17 17:08:52.543543 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.543494    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2"} err="failed to get container status \"1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2\": rpc error: code = NotFound desc = could not find container \"1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2\": container with ID starting with 1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2 not found: ID does not exist"
Sep 17 17:08:52.543543 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.543510    2565 scope.go:117] "RemoveContainer" containerID="740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952"
Sep 17 17:08:52.543735 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:08:52.543721    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952\": container with ID starting with 740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952 not found: ID does not exist" containerID="740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952"
Sep 17 17:08:52.543775 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.543738    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952"} err="failed to get container status \"740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952\": rpc error: code = NotFound desc = could not find container \"740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952\": container with ID starting with 740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952 not found: ID does not exist"
Sep 17 17:08:52.543775 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.543749    2565 scope.go:117] "RemoveContainer" containerID="1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2"
Sep 17 17:08:52.544006 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.543988    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2"} err="failed to get container status \"1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2\": rpc error: code = NotFound desc = could not find container \"1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2\": container with ID starting with 1ef1353e67f35a4422aaf4e25c7107d6aee529dcdf12929a77b970e4f95a75c2 not found: ID does not exist"
Sep 17 17:08:52.544006 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.544006    2565 scope.go:117] "RemoveContainer" containerID="740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952"
Sep 17 17:08:52.544192 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.544177    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952"} err="failed to get container status \"740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952\": rpc error: code = NotFound desc = could not find container \"740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952\": container with ID starting with 740a8f23452f70bbebf1d3e0227ad3243c4008629a5d9a26f415671429bb4952 not found: ID does not exist"
Sep 17 17:08:52.548127 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.548105    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"]
Sep 17 17:08:52.552733 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.552713    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-8dskw"]
Sep 17 17:08:52.601336 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.601314    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:08:52.601336 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.601334    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k45l8\" (UniqueName: \"kubernetes.io/projected/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-kube-api-access-k45l8\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:08:52.601456 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.601344    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd9ec3df-30f4-47e5-95fb-e2647d88bef2-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:08:52.823650 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:52.823592    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd9ec3df-30f4-47e5-95fb-e2647d88bef2" path="/var/lib/kubelet/pods/fd9ec3df-30f4-47e5-95fb-e2647d88bef2/volumes"
Sep 17 17:08:53.771342 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.767690    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x48x6/must-gather-5khbm"]
Sep 17 17:08:53.771342 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.768987    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd9ec3df-30f4-47e5-95fb-e2647d88bef2" containerName="kserve-container"
Sep 17 17:08:53.771342 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.769026    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd9ec3df-30f4-47e5-95fb-e2647d88bef2" containerName="kserve-container"
Sep 17 17:08:53.771342 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.769057    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd9ec3df-30f4-47e5-95fb-e2647d88bef2" containerName="kube-rbac-proxy"
Sep 17 17:08:53.771342 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.769066    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd9ec3df-30f4-47e5-95fb-e2647d88bef2" containerName="kube-rbac-proxy"
Sep 17 17:08:53.771342 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.769263    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd9ec3df-30f4-47e5-95fb-e2647d88bef2" containerName="kube-rbac-proxy"
Sep 17 17:08:53.771342 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.769277    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd9ec3df-30f4-47e5-95fb-e2647d88bef2" containerName="kserve-container"
Sep 17 17:08:53.775079 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.775054    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x48x6/must-gather-5khbm"
Sep 17 17:08:53.777936 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.777897    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x48x6/must-gather-5khbm"]
Sep 17 17:08:53.778869 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.778841    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x48x6\"/\"default-dockercfg-j26dl\""
Sep 17 17:08:53.779005 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.778841    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x48x6\"/\"kube-root-ca.crt\""
Sep 17 17:08:53.779005 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.778887    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x48x6\"/\"openshift-service-ca.crt\""
Sep 17 17:08:53.810145 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.810117    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5fd57759-d597-4f9c-8fc8-a532f419f561-must-gather-output\") pod \"must-gather-5khbm\" (UID: \"5fd57759-d597-4f9c-8fc8-a532f419f561\") " pod="openshift-must-gather-x48x6/must-gather-5khbm"
Sep 17 17:08:53.810265 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.810146    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq4sn\" (UniqueName: \"kubernetes.io/projected/5fd57759-d597-4f9c-8fc8-a532f419f561-kube-api-access-lq4sn\") pod \"must-gather-5khbm\" (UID: \"5fd57759-d597-4f9c-8fc8-a532f419f561\") " pod="openshift-must-gather-x48x6/must-gather-5khbm"
Sep 17 17:08:53.873023 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.872988    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:08:53.911108 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.911079    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5fd57759-d597-4f9c-8fc8-a532f419f561-must-gather-output\") pod \"must-gather-5khbm\" (UID: \"5fd57759-d597-4f9c-8fc8-a532f419f561\") " pod="openshift-must-gather-x48x6/must-gather-5khbm"
Sep 17 17:08:53.911237 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.911111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq4sn\" (UniqueName: \"kubernetes.io/projected/5fd57759-d597-4f9c-8fc8-a532f419f561-kube-api-access-lq4sn\") pod \"must-gather-5khbm\" (UID: \"5fd57759-d597-4f9c-8fc8-a532f419f561\") " pod="openshift-must-gather-x48x6/must-gather-5khbm"
Sep 17 17:08:53.911441 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.911421    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5fd57759-d597-4f9c-8fc8-a532f419f561-must-gather-output\") pod \"must-gather-5khbm\" (UID: \"5fd57759-d597-4f9c-8fc8-a532f419f561\") " pod="openshift-must-gather-x48x6/must-gather-5khbm"
Sep 17 17:08:53.918295 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:53.918268    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq4sn\" (UniqueName: \"kubernetes.io/projected/5fd57759-d597-4f9c-8fc8-a532f419f561-kube-api-access-lq4sn\") pod \"must-gather-5khbm\" (UID: \"5fd57759-d597-4f9c-8fc8-a532f419f561\") " pod="openshift-must-gather-x48x6/must-gather-5khbm"
Sep 17 17:08:54.094203 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:54.094126    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x48x6/must-gather-5khbm"
Sep 17 17:08:54.362456 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:54.362379    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x48x6/must-gather-5khbm"]
Sep 17 17:08:54.365337 ip-10-0-132-85 kubenswrapper[2565]: W0917 17:08:54.365310    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fd57759_d597_4f9c_8fc8_a532f419f561.slice/crio-bc89364ad9fd23c35ef55fa67c859e3fbb14d4036e119dd3898c6f294b2bbe52 WatchSource:0}: Error finding container bc89364ad9fd23c35ef55fa67c859e3fbb14d4036e119dd3898c6f294b2bbe52: Status 404 returned error can't find the container with id bc89364ad9fd23c35ef55fa67c859e3fbb14d4036e119dd3898c6f294b2bbe52
Sep 17 17:08:54.367209 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:54.367192    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:08:54.540390 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:54.540362    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x48x6/must-gather-5khbm" event={"ID":"5fd57759-d597-4f9c-8fc8-a532f419f561","Type":"ContainerStarted","Data":"bc89364ad9fd23c35ef55fa67c859e3fbb14d4036e119dd3898c6f294b2bbe52"}
Sep 17 17:08:55.189543 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:55.189254    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 17:08:58.566002 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:58.565960    2565 generic.go:356] "Generic (PLEG): container finished" podID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerID="e88a3f60c1db5b8026601e2e4d4c6fead5c1e6a9bc05001ede54e4987c2066d9" exitCode=0
Sep 17 17:08:58.566507 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:08:58.566035    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" event={"ID":"5c96fbed-a557-4494-b2f6-e79754072b6b","Type":"ContainerDied","Data":"e88a3f60c1db5b8026601e2e4d4c6fead5c1e6a9bc05001ede54e4987c2066d9"}
Sep 17 17:09:00.188871 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:00.188824    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 17:09:00.193566 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:00.193537    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:09:00.194788 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:00.194762    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 17:09:00.577578 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:00.577547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x48x6/must-gather-5khbm" event={"ID":"5fd57759-d597-4f9c-8fc8-a532f419f561","Type":"ContainerStarted","Data":"e96ec9f0014479fedd3eddd9221909a4f555a5cd61ef1045ee487a95528488e3"}
Sep 17 17:09:01.584030 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:01.583259    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x48x6/must-gather-5khbm" event={"ID":"5fd57759-d597-4f9c-8fc8-a532f419f561","Type":"ContainerStarted","Data":"dcfabcee0971ac49081f6655863f8d531b5688a5895595100fe5faab75bdfc96"}
Sep 17 17:09:01.598223 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:01.598171    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x48x6/must-gather-5khbm" podStartSLOduration=2.621423841 podStartE2EDuration="8.598153647s" totalImagesPullingTime="5.976729806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:08:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:09:01.597709464 +0000 UTC m=+1607.163821185" watchObservedRunningTime="2026-09-17 17:09:01.598153647 +0000 UTC m=+1607.164265370"
Sep 17 17:09:05.188839 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:05.188793    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 17:09:05.189312 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:05.188996    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:09:08.884056 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:08.884020    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kserve-container/0.log"
Sep 17 17:09:08.907831 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:08.907801    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kube-rbac-proxy/0.log"
Sep 17 17:09:08.917887 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:08.917859    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/agent/0.log"
Sep 17 17:09:08.928155 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:08.928133    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/storage-initializer/0.log"
Sep 17 17:09:09.714617 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:09.714584    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kserve-container/0.log"
Sep 17 17:09:09.725618 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:09.725598    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kube-rbac-proxy/0.log"
Sep 17 17:09:09.733867 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:09.733846    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/agent/0.log"
Sep 17 17:09:09.742660 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:09.742636    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/storage-initializer/0.log"
Sep 17 17:09:10.188601 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:10.188568    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 17:09:10.193433 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:10.193393    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:09:10.194786 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:10.194753    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 17:09:10.520668 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:10.520571    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kserve-container/0.log"
Sep 17 17:09:10.531474 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:10.531446    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kube-rbac-proxy/0.log"
Sep 17 17:09:10.540412 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:10.540390    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/agent/0.log"
Sep 17 17:09:10.549033 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:10.549015    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/storage-initializer/0.log"
Sep 17 17:09:11.319896 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:11.319864    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kserve-container/0.log"
Sep 17 17:09:11.331665 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:11.331640    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kube-rbac-proxy/0.log"
Sep 17 17:09:11.341504 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:11.341461    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/agent/0.log"
Sep 17 17:09:11.351353 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:11.351331    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/storage-initializer/0.log"
Sep 17 17:09:12.139805 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:12.139779    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kserve-container/0.log"
Sep 17 17:09:12.151397 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:12.151377    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kube-rbac-proxy/0.log"
Sep 17 17:09:12.160633 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:12.160615    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/agent/0.log"
Sep 17 17:09:12.170513 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:12.170491    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/storage-initializer/0.log"
Sep 17 17:09:12.974007 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:12.973975    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kserve-container/0.log"
Sep 17 17:09:12.985679 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:12.985654    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kube-rbac-proxy/0.log"
Sep 17 17:09:12.995076 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:12.995048    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/agent/0.log"
Sep 17 17:09:13.004777 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:13.004757    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/storage-initializer/0.log"
Sep 17 17:09:13.778879 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:13.778849    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kserve-container/0.log"
Sep 17 17:09:13.790657 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:13.790628    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kube-rbac-proxy/0.log"
Sep 17 17:09:13.800391 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:13.800366    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/agent/0.log"
Sep 17 17:09:13.810631 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:13.810613    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/storage-initializer/0.log"
Sep 17 17:09:14.584024 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:14.583994    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kserve-container/0.log"
Sep 17 17:09:14.598661 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:14.598634    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kube-rbac-proxy/0.log"
Sep 17 17:09:14.609486 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:14.609464    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/agent/0.log"
Sep 17 17:09:14.620472 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:14.620452    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/storage-initializer/0.log"
Sep 17 17:09:15.189072 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:15.189025    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 17:09:15.394161 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:15.394130    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kserve-container/0.log"
Sep 17 17:09:15.406149 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:15.406124    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kube-rbac-proxy/0.log"
Sep 17 17:09:15.417419 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:15.417396    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/agent/0.log"
Sep 17 17:09:15.427339 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:15.427319    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/storage-initializer/0.log"
Sep 17 17:09:16.192136 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:16.192099    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kserve-container/0.log"
Sep 17 17:09:16.203185 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:16.203163    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kube-rbac-proxy/0.log"
Sep 17 17:09:16.213179 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:16.213136    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/agent/0.log"
Sep 17 17:09:16.222154 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:16.222137    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/storage-initializer/0.log"
Sep 17 17:09:17.032439 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:17.032411    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kserve-container/0.log"
Sep 17 17:09:17.044756 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:17.044731    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kube-rbac-proxy/0.log"
Sep 17 17:09:17.054479 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:17.054457    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/agent/0.log"
Sep 17 17:09:17.064709 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:17.064688    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/storage-initializer/0.log"
Sep 17 17:09:17.874435 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:17.874394    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kserve-container/0.log"
Sep 17 17:09:17.886358 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:17.886336    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kube-rbac-proxy/0.log"
Sep 17 17:09:17.895967 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:17.895946    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/agent/0.log"
Sep 17 17:09:17.905462 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:17.905445    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/storage-initializer/0.log"
Sep 17 17:09:18.733629 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:18.733602    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kserve-container/0.log"
Sep 17 17:09:18.745514 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:18.745490    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kube-rbac-proxy/0.log"
Sep 17 17:09:18.755063 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:18.755035    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/agent/0.log"
Sep 17 17:09:18.765521 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:18.765504    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/storage-initializer/0.log"
Sep 17 17:09:19.613448 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:19.613414    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kserve-container/0.log"
Sep 17 17:09:19.625588 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:19.625566    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kube-rbac-proxy/0.log"
Sep 17 17:09:19.635952 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:19.635933    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/agent/0.log"
Sep 17 17:09:19.645662 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:19.645639    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/storage-initializer/0.log"
Sep 17 17:09:20.189070 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:20.189036    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 17:09:20.192932 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:20.192883    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:09:20.193289 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:20.193267    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:09:20.193948 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:20.193899    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 17:09:20.194081 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:20.194039    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:09:20.501867 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:20.501763    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kserve-container/0.log"
Sep 17 17:09:20.512815 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:20.512793    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/kube-rbac-proxy/0.log"
Sep 17 17:09:20.522131 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:20.522111    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/agent/0.log"
Sep 17 17:09:20.531158 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:20.531129    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q_5c96fbed-a557-4494-b2f6-e79754072b6b/storage-initializer/0.log"
Sep 17 17:09:22.381482 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.381458    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:09:22.458752 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.458729    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c96fbed-a557-4494-b2f6-e79754072b6b-kube-rbac-proxy-sar-config\") pod \"5c96fbed-a557-4494-b2f6-e79754072b6b\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") "
Sep 17 17:09:22.458866 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.458758    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c96fbed-a557-4494-b2f6-e79754072b6b-proxy-tls\") pod \"5c96fbed-a557-4494-b2f6-e79754072b6b\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") "
Sep 17 17:09:22.458866 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.458807    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c96fbed-a557-4494-b2f6-e79754072b6b-kserve-provision-location\") pod \"5c96fbed-a557-4494-b2f6-e79754072b6b\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") "
Sep 17 17:09:22.458985 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.458939    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kthn\" (UniqueName: \"kubernetes.io/projected/5c96fbed-a557-4494-b2f6-e79754072b6b-kube-api-access-4kthn\") pod \"5c96fbed-a557-4494-b2f6-e79754072b6b\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") "
Sep 17 17:09:22.459036 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.459003    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c96fbed-a557-4494-b2f6-e79754072b6b-cabundle-cert\") pod \"5c96fbed-a557-4494-b2f6-e79754072b6b\" (UID: \"5c96fbed-a557-4494-b2f6-e79754072b6b\") "
Sep 17 17:09:22.459115 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.459092    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c96fbed-a557-4494-b2f6-e79754072b6b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c96fbed-a557-4494-b2f6-e79754072b6b" (UID: "5c96fbed-a557-4494-b2f6-e79754072b6b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:22.459161 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.459132    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c96fbed-a557-4494-b2f6-e79754072b6b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c96fbed-a557-4494-b2f6-e79754072b6b" (UID: "5c96fbed-a557-4494-b2f6-e79754072b6b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:09:22.459325 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.459308    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c96fbed-a557-4494-b2f6-e79754072b6b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:09:22.459325 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.459325    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c96fbed-a557-4494-b2f6-e79754072b6b-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:09:22.459434 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.459385    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c96fbed-a557-4494-b2f6-e79754072b6b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c96fbed-a557-4494-b2f6-e79754072b6b" (UID: "5c96fbed-a557-4494-b2f6-e79754072b6b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:09:22.460773 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.460752    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c96fbed-a557-4494-b2f6-e79754072b6b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c96fbed-a557-4494-b2f6-e79754072b6b" (UID: "5c96fbed-a557-4494-b2f6-e79754072b6b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:09:22.461032 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.461015    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c96fbed-a557-4494-b2f6-e79754072b6b-kube-api-access-4kthn" (OuterVolumeSpecName: "kube-api-access-4kthn") pod "5c96fbed-a557-4494-b2f6-e79754072b6b" (UID: "5c96fbed-a557-4494-b2f6-e79754072b6b"). InnerVolumeSpecName "kube-api-access-4kthn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:09:22.560490 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.560439    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c96fbed-a557-4494-b2f6-e79754072b6b-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:09:22.560490 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.560460    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kthn\" (UniqueName: \"kubernetes.io/projected/5c96fbed-a557-4494-b2f6-e79754072b6b-kube-api-access-4kthn\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:09:22.560490 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.560469    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c96fbed-a557-4494-b2f6-e79754072b6b-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:09:22.677100 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.677074    2565 generic.go:356] "Generic (PLEG): container finished" podID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerID="51e090358000e1d605b5bb245e439202c60cf9d8c646ecf3aa53cfe862c6e3a9" exitCode=0
Sep 17 17:09:22.677200 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.677150    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"
Sep 17 17:09:22.677200 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.677169    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" event={"ID":"5c96fbed-a557-4494-b2f6-e79754072b6b","Type":"ContainerDied","Data":"51e090358000e1d605b5bb245e439202c60cf9d8c646ecf3aa53cfe862c6e3a9"}
Sep 17 17:09:22.677381 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.677239    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q" event={"ID":"5c96fbed-a557-4494-b2f6-e79754072b6b","Type":"ContainerDied","Data":"d897366c0c23025fcf63662c0969c8ce2b7110c932fe9da10bb2006f816600eb"}
Sep 17 17:09:22.677381 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.677273    2565 scope.go:117] "RemoveContainer" containerID="51e090358000e1d605b5bb245e439202c60cf9d8c646ecf3aa53cfe862c6e3a9"
Sep 17 17:09:22.679193 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.679170    2565 generic.go:356] "Generic (PLEG): container finished" podID="5fd57759-d597-4f9c-8fc8-a532f419f561" containerID="e96ec9f0014479fedd3eddd9221909a4f555a5cd61ef1045ee487a95528488e3" exitCode=0
Sep 17 17:09:22.679313 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.679220    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x48x6/must-gather-5khbm" event={"ID":"5fd57759-d597-4f9c-8fc8-a532f419f561","Type":"ContainerDied","Data":"e96ec9f0014479fedd3eddd9221909a4f555a5cd61ef1045ee487a95528488e3"}
Sep 17 17:09:22.679462 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.679448    2565 scope.go:117] "RemoveContainer" containerID="e96ec9f0014479fedd3eddd9221909a4f555a5cd61ef1045ee487a95528488e3"
Sep 17 17:09:22.685228 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.685210    2565 scope.go:117] "RemoveContainer" containerID="87d348b7be36aa37e6500124dc61d2d6f45231d138b1d0b9b6512a98df337528"
Sep 17 17:09:22.692186 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.692170    2565 scope.go:117] "RemoveContainer" containerID="e88a3f60c1db5b8026601e2e4d4c6fead5c1e6a9bc05001ede54e4987c2066d9"
Sep 17 17:09:22.699976 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.699959    2565 scope.go:117] "RemoveContainer" containerID="d3f25d1c40e71c60d9ad14a25523cac7731532afe511239240db2c7b7d2b8b7b"
Sep 17 17:09:22.707232 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.707211    2565 scope.go:117] "RemoveContainer" containerID="51e090358000e1d605b5bb245e439202c60cf9d8c646ecf3aa53cfe862c6e3a9"
Sep 17 17:09:22.707512 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:09:22.707492    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e090358000e1d605b5bb245e439202c60cf9d8c646ecf3aa53cfe862c6e3a9\": container with ID starting with 51e090358000e1d605b5bb245e439202c60cf9d8c646ecf3aa53cfe862c6e3a9 not found: ID does not exist" containerID="51e090358000e1d605b5bb245e439202c60cf9d8c646ecf3aa53cfe862c6e3a9"
Sep 17 17:09:22.707580 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.707520    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e090358000e1d605b5bb245e439202c60cf9d8c646ecf3aa53cfe862c6e3a9"} err="failed to get container status \"51e090358000e1d605b5bb245e439202c60cf9d8c646ecf3aa53cfe862c6e3a9\": rpc error: code = NotFound desc = could not find container \"51e090358000e1d605b5bb245e439202c60cf9d8c646ecf3aa53cfe862c6e3a9\": container with ID starting with 51e090358000e1d605b5bb245e439202c60cf9d8c646ecf3aa53cfe862c6e3a9 not found: ID does not exist"
Sep 17 17:09:22.707580 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.707536    2565 scope.go:117] "RemoveContainer" containerID="87d348b7be36aa37e6500124dc61d2d6f45231d138b1d0b9b6512a98df337528"
Sep 17 17:09:22.707779 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:09:22.707762    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d348b7be36aa37e6500124dc61d2d6f45231d138b1d0b9b6512a98df337528\": container with ID starting with 87d348b7be36aa37e6500124dc61d2d6f45231d138b1d0b9b6512a98df337528 not found: ID does not exist" containerID="87d348b7be36aa37e6500124dc61d2d6f45231d138b1d0b9b6512a98df337528"
Sep 17 17:09:22.707827 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.707784    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d348b7be36aa37e6500124dc61d2d6f45231d138b1d0b9b6512a98df337528"} err="failed to get container status \"87d348b7be36aa37e6500124dc61d2d6f45231d138b1d0b9b6512a98df337528\": rpc error: code = NotFound desc = could not find container \"87d348b7be36aa37e6500124dc61d2d6f45231d138b1d0b9b6512a98df337528\": container with ID starting with 87d348b7be36aa37e6500124dc61d2d6f45231d138b1d0b9b6512a98df337528 not found: ID does not exist"
Sep 17 17:09:22.707827 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.707796    2565 scope.go:117] "RemoveContainer" containerID="e88a3f60c1db5b8026601e2e4d4c6fead5c1e6a9bc05001ede54e4987c2066d9"
Sep 17 17:09:22.707942 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.707866    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"]
Sep 17 17:09:22.708185 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:09:22.708165    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88a3f60c1db5b8026601e2e4d4c6fead5c1e6a9bc05001ede54e4987c2066d9\": container with ID starting with e88a3f60c1db5b8026601e2e4d4c6fead5c1e6a9bc05001ede54e4987c2066d9 not found: ID does not exist" containerID="e88a3f60c1db5b8026601e2e4d4c6fead5c1e6a9bc05001ede54e4987c2066d9"
Sep 17 17:09:22.708244 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.708191    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88a3f60c1db5b8026601e2e4d4c6fead5c1e6a9bc05001ede54e4987c2066d9"} err="failed to get container status \"e88a3f60c1db5b8026601e2e4d4c6fead5c1e6a9bc05001ede54e4987c2066d9\": rpc error: code = NotFound desc = could not find container \"e88a3f60c1db5b8026601e2e4d4c6fead5c1e6a9bc05001ede54e4987c2066d9\": container with ID starting with e88a3f60c1db5b8026601e2e4d4c6fead5c1e6a9bc05001ede54e4987c2066d9 not found: ID does not exist"
Sep 17 17:09:22.708244 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.708207    2565 scope.go:117] "RemoveContainer" containerID="d3f25d1c40e71c60d9ad14a25523cac7731532afe511239240db2c7b7d2b8b7b"
Sep 17 17:09:22.708489 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:09:22.708473    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f25d1c40e71c60d9ad14a25523cac7731532afe511239240db2c7b7d2b8b7b\": container with ID starting with d3f25d1c40e71c60d9ad14a25523cac7731532afe511239240db2c7b7d2b8b7b not found: ID does not exist" containerID="d3f25d1c40e71c60d9ad14a25523cac7731532afe511239240db2c7b7d2b8b7b"
Sep 17 17:09:22.708529 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.708494    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f25d1c40e71c60d9ad14a25523cac7731532afe511239240db2c7b7d2b8b7b"} err="failed to get container status \"d3f25d1c40e71c60d9ad14a25523cac7731532afe511239240db2c7b7d2b8b7b\": rpc error: code = NotFound desc = could not find container \"d3f25d1c40e71c60d9ad14a25523cac7731532afe511239240db2c7b7d2b8b7b\": container with ID starting with d3f25d1c40e71c60d9ad14a25523cac7731532afe511239240db2c7b7d2b8b7b not found: ID does not exist"
Sep 17 17:09:22.713502 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.713481    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-64c6cf64bf-dwf5q"]
Sep 17 17:09:22.736585 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.736567    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-x48x6_must-gather-5khbm_5fd57759-d597-4f9c-8fc8-a532f419f561/gather/0.log"
Sep 17 17:09:22.823305 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:22.823242    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" path="/var/lib/kubelet/pods/5c96fbed-a557-4494-b2f6-e79754072b6b/volumes"
Sep 17 17:09:23.162037 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.162013    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:09:23.309952 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.309363    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xngqr/must-gather-rftsw"]
Sep 17 17:09:23.310093 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.310027    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kube-rbac-proxy"
Sep 17 17:09:23.310093 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.310045    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kube-rbac-proxy"
Sep 17 17:09:23.310093 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.310086    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="storage-initializer"
Sep 17 17:09:23.310269 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.310095    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="storage-initializer"
Sep 17 17:09:23.310269 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.310113    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kserve-container"
Sep 17 17:09:23.310269 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.310123    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kserve-container"
Sep 17 17:09:23.310269 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.310142    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="agent"
Sep 17 17:09:23.310269 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.310151    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="agent"
Sep 17 17:09:23.310520 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.310318    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kserve-container"
Sep 17 17:09:23.310520 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.310335    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="kube-rbac-proxy"
Sep 17 17:09:23.310520 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.310358    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c96fbed-a557-4494-b2f6-e79754072b6b" containerName="agent"
Sep 17 17:09:23.315205 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.315181    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xngqr/must-gather-rftsw"
Sep 17 17:09:23.318048 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.318031    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xngqr\"/\"kube-root-ca.crt\""
Sep 17 17:09:23.318182 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.318139    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xngqr\"/\"openshift-service-ca.crt\""
Sep 17 17:09:23.319157 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.319133    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xngqr\"/\"default-dockercfg-zpvgc\""
Sep 17 17:09:23.319609 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.319591    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xngqr/must-gather-rftsw"]
Sep 17 17:09:23.323165 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.322567    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:09:23.423622 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.423559    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:09:23.466636 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.466612    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/14e831fd-efa6-413b-a5cc-a2b186214c05-must-gather-output\") pod \"must-gather-rftsw\" (UID: \"14e831fd-efa6-413b-a5cc-a2b186214c05\") " pod="openshift-must-gather-xngqr/must-gather-rftsw"
Sep 17 17:09:23.466723 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.466644    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ddzx\" (UniqueName: \"kubernetes.io/projected/14e831fd-efa6-413b-a5cc-a2b186214c05-kube-api-access-8ddzx\") pod \"must-gather-rftsw\" (UID: \"14e831fd-efa6-413b-a5cc-a2b186214c05\") " pod="openshift-must-gather-xngqr/must-gather-rftsw"
Sep 17 17:09:23.567922 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.567892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/14e831fd-efa6-413b-a5cc-a2b186214c05-must-gather-output\") pod \"must-gather-rftsw\" (UID: \"14e831fd-efa6-413b-a5cc-a2b186214c05\") " pod="openshift-must-gather-xngqr/must-gather-rftsw"
Sep 17 17:09:23.568037 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.567959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ddzx\" (UniqueName: \"kubernetes.io/projected/14e831fd-efa6-413b-a5cc-a2b186214c05-kube-api-access-8ddzx\") pod \"must-gather-rftsw\" (UID: \"14e831fd-efa6-413b-a5cc-a2b186214c05\") " pod="openshift-must-gather-xngqr/must-gather-rftsw"
Sep 17 17:09:23.568250 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.568230    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/14e831fd-efa6-413b-a5cc-a2b186214c05-must-gather-output\") pod \"must-gather-rftsw\" (UID: \"14e831fd-efa6-413b-a5cc-a2b186214c05\") " pod="openshift-must-gather-xngqr/must-gather-rftsw"
Sep 17 17:09:23.577815 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.577785    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ddzx\" (UniqueName: \"kubernetes.io/projected/14e831fd-efa6-413b-a5cc-a2b186214c05-kube-api-access-8ddzx\") pod \"must-gather-rftsw\" (UID: \"14e831fd-efa6-413b-a5cc-a2b186214c05\") " pod="openshift-must-gather-xngqr/must-gather-rftsw"
Sep 17 17:09:23.625458 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.625439    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xngqr/must-gather-rftsw"
Sep 17 17:09:23.893196 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:23.893172    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xngqr/must-gather-rftsw"]
Sep 17 17:09:23.895730 ip-10-0-132-85 kubenswrapper[2565]: W0917 17:09:23.895706    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14e831fd_efa6_413b_a5cc_a2b186214c05.slice/crio-267f4dbad996bd3cea21d461ffe83d72bc0f7f725ac6cc69a66ba6844debd6b4 WatchSource:0}: Error finding container 267f4dbad996bd3cea21d461ffe83d72bc0f7f725ac6cc69a66ba6844debd6b4: Status 404 returned error can't find the container with id 267f4dbad996bd3cea21d461ffe83d72bc0f7f725ac6cc69a66ba6844debd6b4
Sep 17 17:09:24.691159 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:24.691128    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xngqr/must-gather-rftsw" event={"ID":"14e831fd-efa6-413b-a5cc-a2b186214c05","Type":"ContainerStarted","Data":"267f4dbad996bd3cea21d461ffe83d72bc0f7f725ac6cc69a66ba6844debd6b4"}
Sep 17 17:09:25.700698 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:25.699956    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xngqr/must-gather-rftsw" event={"ID":"14e831fd-efa6-413b-a5cc-a2b186214c05","Type":"ContainerStarted","Data":"686546f488c3c832f5c23389223a0971b4ae7e28c29f1b0a89360e5ff144c5a1"}
Sep 17 17:09:25.700698 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:25.700007    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xngqr/must-gather-rftsw" event={"ID":"14e831fd-efa6-413b-a5cc-a2b186214c05","Type":"ContainerStarted","Data":"b28bf02fbedcfa10105a83eac2dd4ff7e5ba9cd794bb630b7232c3cc6cde17d4"}
Sep 17 17:09:25.716147 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:25.716096    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xngqr/must-gather-rftsw" podStartSLOduration=1.811717257 podStartE2EDuration="2.71608136s" totalImagesPullingTime="904.364103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:09:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:09:25.714778075 +0000 UTC m=+1631.280889798" watchObservedRunningTime="2026-09-17 17:09:25.71608136 +0000 UTC m=+1631.282193082"
Sep 17 17:09:26.195840 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:26.195808    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5llzd_70f3faf3-b150-468f-abb7-40f2d221164e/global-pull-secret-syncer/0.log"
Sep 17 17:09:26.345978 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:26.345947    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log"
Sep 17 17:09:26.460381 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:26.460311    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-85.ec2.internal_b8bce5ac69a7a3fe95f9655d41fae9d1/haproxy/0.log"
Sep 17 17:09:28.148453 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.148415    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-x48x6/must-gather-5khbm"]
Sep 17 17:09:28.149197 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.149165    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-x48x6/must-gather-5khbm" podUID="5fd57759-d597-4f9c-8fc8-a532f419f561" containerName="copy" containerID="cri-o://dcfabcee0971ac49081f6655863f8d531b5688a5895595100fe5faab75bdfc96" gracePeriod=2
Sep 17 17:09:28.151692 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:09:28.151667    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5khbm\" is forbidden: User \"system:node:ip-10-0-132-85.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-x48x6\": no relationship found between node 'ip-10-0-132-85.ec2.internal' and this object" podUID="5fd57759-d597-4f9c-8fc8-a532f419f561" pod="openshift-must-gather-x48x6/must-gather-5khbm"
Sep 17 17:09:28.154283 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.154259    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-x48x6/must-gather-5khbm"]
Sep 17 17:09:28.543592 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.543313    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-x48x6_must-gather-5khbm_5fd57759-d597-4f9c-8fc8-a532f419f561/copy/0.log"
Sep 17 17:09:28.549088 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.549060    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x48x6/must-gather-5khbm"
Sep 17 17:09:28.552188 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:09:28.551216    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5khbm\" is forbidden: User \"system:node:ip-10-0-132-85.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-x48x6\": no relationship found between node 'ip-10-0-132-85.ec2.internal' and this object" podUID="5fd57759-d597-4f9c-8fc8-a532f419f561" pod="openshift-must-gather-x48x6/must-gather-5khbm"
Sep 17 17:09:28.622065 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.622021    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq4sn\" (UniqueName: \"kubernetes.io/projected/5fd57759-d597-4f9c-8fc8-a532f419f561-kube-api-access-lq4sn\") pod \"5fd57759-d597-4f9c-8fc8-a532f419f561\" (UID: \"5fd57759-d597-4f9c-8fc8-a532f419f561\") "
Sep 17 17:09:28.622258 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.622085    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5fd57759-d597-4f9c-8fc8-a532f419f561-must-gather-output\") pod \"5fd57759-d597-4f9c-8fc8-a532f419f561\" (UID: \"5fd57759-d597-4f9c-8fc8-a532f419f561\") "
Sep 17 17:09:28.624135 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.624094    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd57759-d597-4f9c-8fc8-a532f419f561-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "5fd57759-d597-4f9c-8fc8-a532f419f561" (UID: "5fd57759-d597-4f9c-8fc8-a532f419f561"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:09:28.634377 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.626820    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd57759-d597-4f9c-8fc8-a532f419f561-kube-api-access-lq4sn" (OuterVolumeSpecName: "kube-api-access-lq4sn") pod "5fd57759-d597-4f9c-8fc8-a532f419f561" (UID: "5fd57759-d597-4f9c-8fc8-a532f419f561"). InnerVolumeSpecName "kube-api-access-lq4sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:09:28.715778 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.715705    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-x48x6_must-gather-5khbm_5fd57759-d597-4f9c-8fc8-a532f419f561/copy/0.log"
Sep 17 17:09:28.716527 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.716501    2565 generic.go:356] "Generic (PLEG): container finished" podID="5fd57759-d597-4f9c-8fc8-a532f419f561" containerID="dcfabcee0971ac49081f6655863f8d531b5688a5895595100fe5faab75bdfc96" exitCode=143
Sep 17 17:09:28.716779 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.716766    2565 scope.go:117] "RemoveContainer" containerID="dcfabcee0971ac49081f6655863f8d531b5688a5895595100fe5faab75bdfc96"
Sep 17 17:09:28.717042 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.717026    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x48x6/must-gather-5khbm"
Sep 17 17:09:28.730616 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.730586    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq4sn\" (UniqueName: \"kubernetes.io/projected/5fd57759-d597-4f9c-8fc8-a532f419f561-kube-api-access-lq4sn\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:09:28.730790 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.730775    2565 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5fd57759-d597-4f9c-8fc8-a532f419f561-must-gather-output\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 17 17:09:28.732004 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:09:28.731904    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5khbm\" is forbidden: User \"system:node:ip-10-0-132-85.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-x48x6\": no relationship found between node 'ip-10-0-132-85.ec2.internal' and this object" podUID="5fd57759-d597-4f9c-8fc8-a532f419f561" pod="openshift-must-gather-x48x6/must-gather-5khbm"
Sep 17 17:09:28.736227 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:09:28.736191    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5khbm\" is forbidden: User \"system:node:ip-10-0-132-85.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-x48x6\": no relationship found between node 'ip-10-0-132-85.ec2.internal' and this object" podUID="5fd57759-d597-4f9c-8fc8-a532f419f561" pod="openshift-must-gather-x48x6/must-gather-5khbm"
Sep 17 17:09:28.744433 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.743732    2565 scope.go:117] "RemoveContainer" containerID="e96ec9f0014479fedd3eddd9221909a4f555a5cd61ef1045ee487a95528488e3"
Sep 17 17:09:28.768035 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.767891    2565 scope.go:117] "RemoveContainer" containerID="dcfabcee0971ac49081f6655863f8d531b5688a5895595100fe5faab75bdfc96"
Sep 17 17:09:28.768605 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:09:28.768465    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcfabcee0971ac49081f6655863f8d531b5688a5895595100fe5faab75bdfc96\": container with ID starting with dcfabcee0971ac49081f6655863f8d531b5688a5895595100fe5faab75bdfc96 not found: ID does not exist" containerID="dcfabcee0971ac49081f6655863f8d531b5688a5895595100fe5faab75bdfc96"
Sep 17 17:09:28.768605 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.768504    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfabcee0971ac49081f6655863f8d531b5688a5895595100fe5faab75bdfc96"} err="failed to get container status \"dcfabcee0971ac49081f6655863f8d531b5688a5895595100fe5faab75bdfc96\": rpc error: code = NotFound desc = could not find container \"dcfabcee0971ac49081f6655863f8d531b5688a5895595100fe5faab75bdfc96\": container with ID starting with dcfabcee0971ac49081f6655863f8d531b5688a5895595100fe5faab75bdfc96 not found: ID does not exist"
Sep 17 17:09:28.768605 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.768530    2565 scope.go:117] "RemoveContainer" containerID="e96ec9f0014479fedd3eddd9221909a4f555a5cd61ef1045ee487a95528488e3"
Sep 17 17:09:28.769202 ip-10-0-132-85 kubenswrapper[2565]: E0917 17:09:28.769120    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e96ec9f0014479fedd3eddd9221909a4f555a5cd61ef1045ee487a95528488e3\": container with ID starting with e96ec9f0014479fedd3eddd9221909a4f555a5cd61ef1045ee487a95528488e3 not found: ID does not exist" containerID="e96ec9f0014479fedd3eddd9221909a4f555a5cd61ef1045ee487a95528488e3"
Sep 17 17:09:28.769202 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.769153    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e96ec9f0014479fedd3eddd9221909a4f555a5cd61ef1045ee487a95528488e3"} err="failed to get container status \"e96ec9f0014479fedd3eddd9221909a4f555a5cd61ef1045ee487a95528488e3\": rpc error: code = NotFound desc = could not find container \"e96ec9f0014479fedd3eddd9221909a4f555a5cd61ef1045ee487a95528488e3\": container with ID starting with e96ec9f0014479fedd3eddd9221909a4f555a5cd61ef1045ee487a95528488e3 not found: ID does not exist"
Sep 17 17:09:28.846555 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:28.846518    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fd57759-d597-4f9c-8fc8-a532f419f561" path="/var/lib/kubelet/pods/5fd57759-d597-4f9c-8fc8-a532f419f561/volumes"
Sep 17 17:09:29.015116 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:29.014990    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:09:29.071432 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:29.071398    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bn474_3ce775b0-037d-4846-a3b6-f8bf2f76f293/konnectivity-agent/0.log" limit=1024
Sep 17 17:09:30.843760 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:30.843724    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-stxzq_b9d6184f-c101-4444-bcdf-c935a87c813c/kube-state-metrics/0.log"
Sep 17 17:09:30.869166 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:30.869089    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-stxzq_b9d6184f-c101-4444-bcdf-c935a87c813c/kube-rbac-proxy-main/0.log"
Sep 17 17:09:30.889019 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:30.888990    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-stxzq_b9d6184f-c101-4444-bcdf-c935a87c813c/kube-rbac-proxy-self/0.log"
Sep 17 17:09:30.928339 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:30.928308    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6fdd44fc8b-kzwf5_da4f2ef7-b89c-4a07-ad96-188207e336bf/metrics-server/0.log"
Sep 17 17:09:30.952630 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:30.952551    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-tbsbh_95994e0a-0887-4830-8d71-2a2aac599932/monitoring-plugin/0.log"
Sep 17 17:09:31.051325 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.051294    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cq9n8_a62e64e0-3747-4257-ad9e-fb329dd8ac70/node-exporter/0.log"
Sep 17 17:09:31.083145 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.083061    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cq9n8_a62e64e0-3747-4257-ad9e-fb329dd8ac70/kube-rbac-proxy/0.log"
Sep 17 17:09:31.105205 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.105164    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cq9n8_a62e64e0-3747-4257-ad9e-fb329dd8ac70/init-textfile/0.log"
Sep 17 17:09:31.222499 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.222459    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-k4cjr_a691796a-00c8-41d7-8440-bb98d19476f5/kube-rbac-proxy-main/0.log"
Sep 17 17:09:31.248231 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.248152    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-k4cjr_a691796a-00c8-41d7-8440-bb98d19476f5/kube-rbac-proxy-self/0.log"
Sep 17 17:09:31.269543 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.269518    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-k4cjr_a691796a-00c8-41d7-8440-bb98d19476f5/openshift-state-metrics/0.log"
Sep 17 17:09:31.313501 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.313471    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_02f0ef06-ac87-4d72-a9ba-70f8fdbd311a/prometheus/0.log"
Sep 17 17:09:31.331492 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.331459    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_02f0ef06-ac87-4d72-a9ba-70f8fdbd311a/config-reloader/0.log"
Sep 17 17:09:31.355265 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.355189    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_02f0ef06-ac87-4d72-a9ba-70f8fdbd311a/thanos-sidecar/0.log"
Sep 17 17:09:31.376839 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.376809    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_02f0ef06-ac87-4d72-a9ba-70f8fdbd311a/kube-rbac-proxy-web/0.log"
Sep 17 17:09:31.396403 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.396376    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_02f0ef06-ac87-4d72-a9ba-70f8fdbd311a/kube-rbac-proxy/0.log"
Sep 17 17:09:31.416568 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.416536    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_02f0ef06-ac87-4d72-a9ba-70f8fdbd311a/kube-rbac-proxy-thanos/0.log"
Sep 17 17:09:31.440143 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.440107    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_02f0ef06-ac87-4d72-a9ba-70f8fdbd311a/init-config-reloader/0.log"
Sep 17 17:09:31.616984 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.616907    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54576c98d9-dm9pd_d6bee9be-246a-4c3f-8f63-656d809eb7d9/thanos-query/0.log"
Sep 17 17:09:31.637754 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.637686    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54576c98d9-dm9pd_d6bee9be-246a-4c3f-8f63-656d809eb7d9/kube-rbac-proxy-web/0.log"
Sep 17 17:09:31.659126 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.659083    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54576c98d9-dm9pd_d6bee9be-246a-4c3f-8f63-656d809eb7d9/kube-rbac-proxy/0.log"
Sep 17 17:09:31.681387 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.681353    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54576c98d9-dm9pd_d6bee9be-246a-4c3f-8f63-656d809eb7d9/prom-label-proxy/0.log"
Sep 17 17:09:31.704132 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.704087    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54576c98d9-dm9pd_d6bee9be-246a-4c3f-8f63-656d809eb7d9/kube-rbac-proxy-rules/0.log"
Sep 17 17:09:31.724609 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:31.724575    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54576c98d9-dm9pd_d6bee9be-246a-4c3f-8f63-656d809eb7d9/kube-rbac-proxy-metrics/0.log"
Sep 17 17:09:34.234074 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.234038    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"]
Sep 17 17:09:34.234526 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.234461    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd57759-d597-4f9c-8fc8-a532f419f561" containerName="copy"
Sep 17 17:09:34.234526 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.234476    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd57759-d597-4f9c-8fc8-a532f419f561" containerName="copy"
Sep 17 17:09:34.234526 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.234508    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd57759-d597-4f9c-8fc8-a532f419f561" containerName="gather"
Sep 17 17:09:34.234526 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.234516    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd57759-d597-4f9c-8fc8-a532f419f561" containerName="gather"
Sep 17 17:09:34.234661 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.234578    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd57759-d597-4f9c-8fc8-a532f419f561" containerName="copy"
Sep 17 17:09:34.234661 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.234592    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd57759-d597-4f9c-8fc8-a532f419f561" containerName="gather"
Sep 17 17:09:34.238271 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.238244    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.247806 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.247779    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"]
Sep 17 17:09:34.391970 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.391939    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d44c1a80-32be-400c-b731-23606fc5a050-podres\") pod \"perf-node-gather-daemonset-pgnzp\" (UID: \"d44c1a80-32be-400c-b731-23606fc5a050\") " pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.391970 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.391976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d44c1a80-32be-400c-b731-23606fc5a050-proc\") pod \"perf-node-gather-daemonset-pgnzp\" (UID: \"d44c1a80-32be-400c-b731-23606fc5a050\") " pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.392196 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.391997    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d44c1a80-32be-400c-b731-23606fc5a050-sys\") pod \"perf-node-gather-daemonset-pgnzp\" (UID: \"d44c1a80-32be-400c-b731-23606fc5a050\") " pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.392196 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.392108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nkdn\" (UniqueName: \"kubernetes.io/projected/d44c1a80-32be-400c-b731-23606fc5a050-kube-api-access-8nkdn\") pod \"perf-node-gather-daemonset-pgnzp\" (UID: \"d44c1a80-32be-400c-b731-23606fc5a050\") " pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.392196 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.392171    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d44c1a80-32be-400c-b731-23606fc5a050-lib-modules\") pod \"perf-node-gather-daemonset-pgnzp\" (UID: \"d44c1a80-32be-400c-b731-23606fc5a050\") " pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.493416 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.493331    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nkdn\" (UniqueName: \"kubernetes.io/projected/d44c1a80-32be-400c-b731-23606fc5a050-kube-api-access-8nkdn\") pod \"perf-node-gather-daemonset-pgnzp\" (UID: \"d44c1a80-32be-400c-b731-23606fc5a050\") " pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.493701 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.493675    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d44c1a80-32be-400c-b731-23606fc5a050-lib-modules\") pod \"perf-node-gather-daemonset-pgnzp\" (UID: \"d44c1a80-32be-400c-b731-23606fc5a050\") " pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.493869 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.493852    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d44c1a80-32be-400c-b731-23606fc5a050-podres\") pod \"perf-node-gather-daemonset-pgnzp\" (UID: \"d44c1a80-32be-400c-b731-23606fc5a050\") " pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.494054 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.494032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d44c1a80-32be-400c-b731-23606fc5a050-proc\") pod \"perf-node-gather-daemonset-pgnzp\" (UID: \"d44c1a80-32be-400c-b731-23606fc5a050\") " pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.494169 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.494153    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d44c1a80-32be-400c-b731-23606fc5a050-sys\") pod \"perf-node-gather-daemonset-pgnzp\" (UID: \"d44c1a80-32be-400c-b731-23606fc5a050\") " pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.494379 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.494363    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d44c1a80-32be-400c-b731-23606fc5a050-sys\") pod \"perf-node-gather-daemonset-pgnzp\" (UID: \"d44c1a80-32be-400c-b731-23606fc5a050\") " pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.494933 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.494891    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d44c1a80-32be-400c-b731-23606fc5a050-lib-modules\") pod \"perf-node-gather-daemonset-pgnzp\" (UID: \"d44c1a80-32be-400c-b731-23606fc5a050\") " pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.495164 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.495146    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d44c1a80-32be-400c-b731-23606fc5a050-podres\") pod \"perf-node-gather-daemonset-pgnzp\" (UID: \"d44c1a80-32be-400c-b731-23606fc5a050\") " pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.495305 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.495291    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d44c1a80-32be-400c-b731-23606fc5a050-proc\") pod \"perf-node-gather-daemonset-pgnzp\" (UID: \"d44c1a80-32be-400c-b731-23606fc5a050\") " pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.502399 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.502373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nkdn\" (UniqueName: \"kubernetes.io/projected/d44c1a80-32be-400c-b731-23606fc5a050-kube-api-access-8nkdn\") pod \"perf-node-gather-daemonset-pgnzp\" (UID: \"d44c1a80-32be-400c-b731-23606fc5a050\") " pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.550514 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.550480    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:34.834167 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:34.834138    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"]
Sep 17 17:09:35.411463 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:35.411426    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-28z7f_61ab0f21-2b0c-4e87-9887-8bbc281cf2d5/dns/0.log"
Sep 17 17:09:35.431976 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:35.431947    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-28z7f_61ab0f21-2b0c-4e87-9887-8bbc281cf2d5/kube-rbac-proxy/0.log"
Sep 17 17:09:35.552895 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:35.552865    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dtmtt_4a45b30c-1f89-4643-84b2-189a36d6e0f8/dns-node-resolver/0.log"
Sep 17 17:09:35.760609 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:35.760521    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp" event={"ID":"d44c1a80-32be-400c-b731-23606fc5a050","Type":"ContainerStarted","Data":"7194b97c1eaaba6d0953c2bde07a3fc8167d118a4f9cbae6e75ba71a051aa6da"}
Sep 17 17:09:35.760609 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:35.760588    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp" event={"ID":"d44c1a80-32be-400c-b731-23606fc5a050","Type":"ContainerStarted","Data":"00c289298e4a14ab89dbd3ae7853957dc7c7a80b37bb5f60f32668d2555c7251"}
Sep 17 17:09:35.760812 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:35.760618    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:35.778988 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:35.778928    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp" podStartSLOduration=1.778891529 podStartE2EDuration="1.778891529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:09:35.776384407 +0000 UTC m=+1641.342496154" watchObservedRunningTime="2026-09-17 17:09:35.778891529 +0000 UTC m=+1641.345003252"
Sep 17 17:09:36.070625 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:36.070544    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-69bv6_307aa194-969e-445a-8f44-22359d43a8b5/node-ca/0.log"
Sep 17 17:09:37.170403 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:37.170370    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7h2vr_a356edd2-b0ce-4622-be5a-eb1062dfef07/serve-healthcheck-canary/0.log"
Sep 17 17:09:37.638271 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:37.638236    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8l5mv_7274b4f9-685c-4c23-abb9-4c6f88fcdcc7/kube-rbac-proxy/0.log"
Sep 17 17:09:37.656451 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:37.656424    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8l5mv_7274b4f9-685c-4c23-abb9-4c6f88fcdcc7/exporter/0.log"
Sep 17 17:09:37.675531 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:37.675503    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8l5mv_7274b4f9-685c-4c23-abb9-4c6f88fcdcc7/extractor/0.log"
Sep 17 17:09:40.336312 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:40.336281    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-9sx8z_186d3f5e-1136-43b6-a41a-4750dd3e1a65/manager/0.log"
Sep 17 17:09:40.378443 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:40.378413    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-mq9nl_5e860760-2ab7-4db2-80af-91def4cdc1dd/s3-init/0.log"
Sep 17 17:09:40.393182 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:40.393158    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-mq9nl_5e860760-2ab7-4db2-80af-91def4cdc1dd/download-hf-model/0.log"
Sep 17 17:09:40.420467 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:40.420446    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-6tlkz_da80d565-728c-4e46-b0a8-330443915b96/seaweedfs/0.log"
Sep 17 17:09:41.780366 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:41.779798    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xngqr/perf-node-gather-daemonset-pgnzp"
Sep 17 17:09:45.979398 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:45.979370    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qvxw9_c4b8badb-da9b-4399-96ad-aa05263c9542/kube-multus-additional-cni-plugins/0.log"
Sep 17 17:09:45.998722 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:45.998691    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qvxw9_c4b8badb-da9b-4399-96ad-aa05263c9542/egress-router-binary-copy/0.log"
Sep 17 17:09:46.018185 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:46.018162    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qvxw9_c4b8badb-da9b-4399-96ad-aa05263c9542/cni-plugins/0.log"
Sep 17 17:09:46.036826 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:46.036804    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qvxw9_c4b8badb-da9b-4399-96ad-aa05263c9542/bond-cni-plugin/0.log"
Sep 17 17:09:46.055834 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:46.055810    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qvxw9_c4b8badb-da9b-4399-96ad-aa05263c9542/routeoverride-cni/0.log"
Sep 17 17:09:46.075391 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:46.075374    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qvxw9_c4b8badb-da9b-4399-96ad-aa05263c9542/whereabouts-cni-bincopy/0.log"
Sep 17 17:09:46.095282 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:46.095258    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qvxw9_c4b8badb-da9b-4399-96ad-aa05263c9542/whereabouts-cni/0.log"
Sep 17 17:09:46.347621 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:46.347564    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-s5lfc_aef9115b-1284-4e79-a8e3-9e8d1778a76e/kube-multus/0.log"
Sep 17 17:09:46.367774 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:46.367751    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wld7n_716feac9-5ed2-4b28-9ede-a210a4871e3d/network-metrics-daemon/0.log"
Sep 17 17:09:46.383320 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:46.383302    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wld7n_716feac9-5ed2-4b28-9ede-a210a4871e3d/kube-rbac-proxy/0.log"
Sep 17 17:09:47.584014 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:47.583962    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q7v29_700d8b01-1d3b-4b49-bb6a-696687e029bb/ovn-controller/0.log"
Sep 17 17:09:47.598709 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:47.598688    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q7v29_700d8b01-1d3b-4b49-bb6a-696687e029bb/ovn-acl-logging/0.log"
Sep 17 17:09:47.619584 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:47.619557    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q7v29_700d8b01-1d3b-4b49-bb6a-696687e029bb/kube-rbac-proxy-node/0.log"
Sep 17 17:09:47.638445 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:47.638425    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q7v29_700d8b01-1d3b-4b49-bb6a-696687e029bb/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 17:09:47.657799 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:47.657780    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q7v29_700d8b01-1d3b-4b49-bb6a-696687e029bb/northd/0.log"
Sep 17 17:09:47.676272 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:47.676249    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q7v29_700d8b01-1d3b-4b49-bb6a-696687e029bb/nbdb/0.log"
Sep 17 17:09:47.695852 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:47.695828    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q7v29_700d8b01-1d3b-4b49-bb6a-696687e029bb/sbdb/0.log"
Sep 17 17:09:47.751438 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:47.751367    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-q7v29_700d8b01-1d3b-4b49-bb6a-696687e029bb/ovnkube-controller/0.log"
Sep 17 17:09:48.709964 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:48.709909    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-zl97d_c0a61d36-0e50-4539-ac3f-d5b6c9c3e287/network-check-target-container/0.log"
Sep 17 17:09:49.563273 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:49.563241    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-6hz2q_8a6a6976-08a4-4f97-9abe-a558269301a0/iptables-alerter/0.log"
Sep 17 17:09:50.281132 ip-10-0-132-85 kubenswrapper[2565]: I0917 17:09:50.281104    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-cqx8k_ffd47162-fa3a-4c51-a370-3ffe004cf33a/tuned/0.log"