Sep 11 16:14:48.329220 ip-10-0-142-248 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 16:14:48.329229 ip-10-0-142-248 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 16:14:48.329237 ip-10-0-142-248 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 16:14:48.329453 ip-10-0-142-248 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 16:14:57.608271 ip-10-0-142-248 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 16:14:57.608283 ip-10-0-142-248 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a0e22abc5f724d6aa13b5823477ce807 --
Sep 11 16:16:33.340437 ip-10-0-142-248 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 16:16:33.774583 ip-10-0-142-248 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:16:33.774583 ip-10-0-142-248 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 16:16:33.774583 ip-10-0-142-248 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:16:33.774583 ip-10-0-142-248 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 16:16:33.774583 ip-10-0-142-248 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:16:33.775769 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.775645    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 16:16:33.779806 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779789    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:33.779806 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779806    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779810    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779814    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779817    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779820    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779823    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779826    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779828    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779831    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779834    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779836    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779839    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779842    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779846    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779850    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779853    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779857    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779860    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779862    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:33.779873 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779865    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779868    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779871    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779873    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779877    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779879    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779885    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779888    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779891    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779894    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779897    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779900    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779903    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779905    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779908    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779911    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779913    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779916    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779918    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779921    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:33.780343 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779923    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779926    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779928    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779931    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779933    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779936    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779938    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779941    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779943    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779946    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779948    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779951    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779959    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779961    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779965    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779967    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779970    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779972    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779975    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779978    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:33.780821 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779980    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779983    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779986    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779988    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779991    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779993    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779995    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.779998    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780001    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780003    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780006    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780008    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780011    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780013    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780016    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780018    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780021    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780024    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780026    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780029    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:33.781374 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780031    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780034    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780037    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780041    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780045    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780047    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780051    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780054    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780057    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780059    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780062    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780065    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780068    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780070    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780073    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780075    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780078    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780080    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:33.781863 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.780083    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:33.782881 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782868    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:33.782881 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782881    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782884    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782888    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782891    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782894    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782896    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782899    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782901    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782904    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782908    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782919    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782922    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782925    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782928    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782931    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782933    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782936    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782939    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:33.782936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782943    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782947    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782951    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782953    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782956    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782959    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782961    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782964    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782966    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782969    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782972    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782976    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782978    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782982    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782984    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782987    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782989    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782992    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782994    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:33.783391 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782996    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.782999    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783002    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783004    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783006    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783009    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783017    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783020    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783022    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783025    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783027    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783030    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783032    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783035    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783037    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783041    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783043    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783046    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783048    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783051    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:33.783860 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783054    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783057    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783059    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783062    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783065    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783068    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783070    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783073    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783075    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783078    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783080    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783083    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783085    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783088    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783090    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783093    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783095    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783097    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783100    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783108    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:33.784379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783111    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783113    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783115    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783118    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783121    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783123    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783126    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783133    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783136    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783138    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783141    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783143    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783146    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783148    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783151    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783153    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783156    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783158    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783173    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:33.784874 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783176    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783179    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783582    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783588    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783590    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783593    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783596    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783599    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783603    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783607    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783610    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783613    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783617    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783620    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783623    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783626    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783628    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783631    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783634    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:33.785349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783636    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783641    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783643    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783646    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783649    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783653    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783656    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783659    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783662    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783665    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783668    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783672    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783675    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783678    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783680    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783683    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783686    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783688    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783691    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:33.785807 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783693    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783696    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783698    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783701    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783704    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783706    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783709    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783712    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783715    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783717    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783720    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783723    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783725    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783728    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783732    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783734    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783737    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783739    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783742    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783745    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:33.786286 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783747    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783750    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783752    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783755    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783758    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783760    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783763    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783765    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783768    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783771    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783774    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783776    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783779    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783782    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783784    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783787    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783789    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783793    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783796    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783798    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:33.786800 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783801    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783804    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783806    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783809    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783811    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783814    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783818    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783820    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783823    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783825    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783828    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783831    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783834    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783836    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783840    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783843    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783846    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783849    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783851    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783854    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:33.787295 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783857    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783859    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.783862    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.783956    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.783995    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784002    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784007    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784012    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784015    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784020    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784025    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784028    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784032    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784038    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784042    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784045    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784048    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784051    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784054    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784058    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784061    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784065    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784068    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 16:16:33.787777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784071    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784075    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784078    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784082    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784085    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784088    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784091    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784094    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784097    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784100    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784103    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784106    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784111    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784114    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784117    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784120    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784123    2561 flags.go:64] FLAG: --enable-server="true"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784126    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784131    2561 flags.go:64] FLAG: --event-burst="100"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784134    2561 flags.go:64] FLAG: --event-qps="50"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784138    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784141    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784144    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784148    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 16:16:33.788335 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784151    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784154    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784157    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784178    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784182    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784185    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784188    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784191    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784194    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784197    2561 flags.go:64] FLAG: --feature-gates=""
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784201    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784204    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784207    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784210    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784213    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784216    2561 flags.go:64] FLAG: --help="false"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784219    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-142-248.ec2.internal"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784223    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784227    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784230    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784233    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784237    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784240    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 16:16:33.788910 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784243    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784246    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784248    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784252    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784255    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784258    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784261    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784264    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784267    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784271    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784274    2561 flags.go:64] FLAG: --lock-file=""
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784276    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784279    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784282    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784288    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784291    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784293    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784296    2561 flags.go:64] FLAG: --logging-format="text"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784299    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784302    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784305    2561 flags.go:64] FLAG: --manifest-url=""
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784308    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784312    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784315    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784319    2561 flags.go:64] FLAG: --max-pods="110"
Sep 11 16:16:33.789519 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784322    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784325    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784329    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784332    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784335    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784338    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784341    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784347    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784350    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784353    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784356    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784360    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784364    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784367    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784371    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784373    2561 flags.go:64] FLAG: --port="10250"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784376    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784380    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02498e9657d229644"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784384    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784387    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784390    2561 flags.go:64] FLAG: --register-node="true"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784393    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784397    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784400    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 11 16:16:33.790182 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784403    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784406    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784409    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784413    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784416    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784419    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784421    2561 flags.go:64] FLAG: --runonce="false"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784424    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784428    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784430    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784435    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784438    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784441    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784444    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784447    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784450    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784453    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784456    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784458    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784462    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784465    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784467    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784473    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784476    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784479    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 16:16:33.790764 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784483    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784486    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784489    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784493    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784496    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784499    2561 flags.go:64] FLAG: --v="2"
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784503    2561 flags.go:64] FLAG: --version="false"
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784506    2561 flags.go:64] FLAG: --vmodule=""
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784511    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.784514    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784628    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784633    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784636    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784639    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784642    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784645    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784648    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784653    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784656    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784659    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784662    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784664    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:33.791379 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784667    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784670    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784673    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784676    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784678    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784681    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784684    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784686    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784689    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784691    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784694    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784696    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784699    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784703    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784706    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784709    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784711    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784714    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784717    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:33.791923 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784719    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784722    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784725    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784728    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784730    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784733    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784735    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784738    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784742    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784745    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784747    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784750    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784752    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784756    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784760    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784763    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784765    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784768    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784771    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:33.792418 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784774    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784776    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784779    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784781    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784785    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784788    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784791    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784797    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784800    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784803    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784806    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784809    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784812    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784815    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784818    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784821    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784824    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784826    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784829    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:33.792882 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784832    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784835    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784838    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784841    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784843    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784846    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784848    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784852    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784854    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784856    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784859    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784861    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784864    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784866    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784869    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784871    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784874    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784877    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784880    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784883    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:33.793437 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784887    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:33.793925 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784897    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:33.793925 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784900    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:33.793925 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784903    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:33.793925 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784905    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:33.793925 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784908    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:33.793925 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784910    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:33.793925 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784913    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:33.793925 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784915    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:33.793925 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.784918    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:33.793925 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.785840    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:16:33.793925 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.793712    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 16:16:33.793925 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.793840    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 16:16:33.794779 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.794768    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 16:16:33.794811 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.794779    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 16:16:33.794847 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794839    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:33.794847 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794846    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794850    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794854    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794857    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794860    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794862    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794866    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794871    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794874    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794877    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794880    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794883    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794886    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794889    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794892    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794894    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794897    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794900    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794903    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:33.794901 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794906    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794910    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794913    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794917    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794920    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794923    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794926    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794929    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794932    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794935    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794938    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794940    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794943    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794945    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794948    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794950    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794953    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794955    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794958    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:33.795450 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794960    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794963    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794965    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794968    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794970    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794973    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794975    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794978    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794981    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794983    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794986    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794988    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794991    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794994    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794996    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.794999    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795002    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795005    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795007    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:33.795893 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795010    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795013    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795015    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795018    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795021    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795024    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795027    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795029    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795032    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795035    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795037    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795040    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795043    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795045    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795048    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795050    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795053    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795056    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795059    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795061    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:33.796360 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795064    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795066    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795069    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795071    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795075    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795078    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795081    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795084    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795087    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795090    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795093    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795095    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795098    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795100    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795103    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795105    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795108    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795111    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795113    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:33.796856 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795116    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:33.797349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795119    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:33.797349 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.795124    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:16:33.797349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795249    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:33.797349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795254    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:33.797349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795257    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:33.797349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795260    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:33.797349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795264    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:33.797349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795268    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:33.797349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795271    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:33.797349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795274    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:33.797349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795276    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:33.797349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795279    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:33.797349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795282    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:33.797349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795285    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:33.797349 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795288    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795290    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795292    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795295    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795298    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795301    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795303    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795306    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795308    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795311    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795314    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795317    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795319    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795322    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795325    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795327    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795331    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795333    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795336    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795338    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:33.797710 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795341    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795344    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795346    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795349    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795351    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795355    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795358    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795361    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795364    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795366    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795369    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795372    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795374    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795377    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795379    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795382    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795385    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795387    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795390    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:33.798202 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795392    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795395    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795398    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795400    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795403    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795406    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795416    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795419    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795422    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795425    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795428    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795431    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795434    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795436    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795440    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795442    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795445    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795447    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795450    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795453    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:33.798664 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795456    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795459    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795462    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795464    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795467    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795470    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795473    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795476    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795479    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795482    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795485    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795487    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795490    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795493    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795496    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795499    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795501    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795504    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795507    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:33.799140 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795509    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:33.799700 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795512    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:33.799700 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795514    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:33.799700 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795517    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:33.799700 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795520    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:33.799700 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795522    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:33.799700 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795525    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:33.799700 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795527    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:33.799700 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:33.795530    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:33.799700 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.795535    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:16:33.799700 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.795681    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 16:16:33.799700 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.799210    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 16:16:33.800387 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.800376    2561 server.go:1019] "Starting client certificate rotation"
Sep 11 16:16:33.800484 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.800471    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 16:16:33.800526 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.800505    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 16:16:33.805156 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.805125    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 16:16:33.809114 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.809098    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 16:16:33.820085 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.820062    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 11 16:16:33.824308 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.824293    2561 log.go:25] "Validated CRI v1 image API"
Sep 11 16:16:33.826800 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.826783    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 16:16:33.830200 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.830160    2561 fs.go:135] Filesystem UUIDs: map[60e761a8-b48b-4139-8fe0-d85ec47ca9fc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b3a3a8c1-dcf8-47dd-bf7a-53426a9f5ab2:/dev/nvme0n1p4]
Sep 11 16:16:33.830275 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.830201    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 16:16:33.835520 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.835395    2561 manager.go:216] Machine: {Timestamp:2026-09-11 16:16:33.833996197 +0000 UTC m=+0.379100039 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100160 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23550f43e252e09764717a5c1c47c2 SystemUUID:ec23550f-43e2-52e0-9764-717a5c1c47c2 BootID:a0e22abc-5f72-4d6a-a13b-5823477ce807 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c7:ba:2c:a1:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c7:ba:2c:a1:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:24:f0:4d:d1:e3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 16:16:33.835520 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.835518    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 16:16:33.835654 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.835642    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 16:16:33.839244 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.839213    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 16:16:33.839389 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.839247    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-248.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 16:16:33.840206 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.840185    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 16:16:33.840206 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.840208    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 16:16:33.840282 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.840217    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 16:16:33.840282 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.840231    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 16:16:33.841500 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.841489    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 16:16:33.846246 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.846233    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 16:16:33.846377 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.846367    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 16:16:33.846416 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.846407    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 16:16:33.846447 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.846426    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 16:16:33.846447 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.846442    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 16:16:33.846506 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.846452    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 11 16:16:33.846506 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.846463    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 16:16:33.846856 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.846844    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 16:16:33.846897 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.846864    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 16:16:33.849348 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.849330    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 16:16:33.851009 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.850991    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 16:16:33.851819 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.851803    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 16:16:33.851905 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.851842    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 16:16:33.854672 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.854656    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 16:16:33.854751 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.854681    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 16:16:33.854751 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.854690    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 16:16:33.854751 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.854698    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 16:16:33.854751 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.854742    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 16:16:33.854751 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.854752    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 16:16:33.854942 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.854771    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 16:16:33.854942 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.854779    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 16:16:33.854942 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.854789    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 16:16:33.854942 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.854797    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 16:16:33.854942 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.854816    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 16:16:33.854942 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.854825    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 16:16:33.855725 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.855714    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 16:16:33.855776 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.855729    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 16:16:33.859778 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.859758    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-248.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:16:33.859928 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:33.859906    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 16:16:33.860001 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:33.859978    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 16:16:33.860815 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.860799    2561 server.go:1268] "Started kubelet"
Sep 11 16:16:33.860899 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.860868    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 16:16:33.861009 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.860956    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 16:16:33.861125 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.861028    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 16:16:33.861833 ip-10-0-142-248 systemd[1]: Started Kubernetes Kubelet.
Sep 11 16:16:33.863092 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.862896    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 16:16:33.865340 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:33.865318    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 16:16:33.870735 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:33.869558    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-248.ec2.internal.18d4506ffb4ec60a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-248.ec2.internal,UID:ip-10-0-142-248.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-248.ec2.internal,},FirstTimestamp:2026-09-11 16:16:33.860765194 +0000 UTC m=+0.405869034,LastTimestamp:2026-09-11 16:16:33.860765194 +0000 UTC m=+0.405869034,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-248.ec2.internal,}"
Sep 11 16:16:33.914755 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.914728    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4wpvf"
Sep 11 16:16:33.925962 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.925938    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4wpvf"
Sep 11 16:16:33.997975 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:33.997950    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 16:16:34.005207 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.005183    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 16:16:34.005842 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.005826    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 16:16:34.006011 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:34.005992    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-248.ec2.internal\" not found"
Sep 11 16:16:34.006700 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.006680    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 16:16:34.006700 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.006686    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 16:16:34.006831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.006710    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 16:16:34.006870 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.006855    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 16:16:34.006870 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.006866    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 16:16:34.007120 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.007103    2561 factory.go:153] Registering CRI-O factory
Sep 11 16:16:34.007181 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.007125    2561 factory.go:223] Registration of the crio container factory successfully
Sep 11 16:16:34.007223 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.007211    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 16:16:34.007261 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.007225    2561 factory.go:55] Registering systemd factory
Sep 11 16:16:34.007261 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.007234    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 11 16:16:34.007261 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.007251    2561 factory.go:103] Registering Raw factory
Sep 11 16:16:34.007362 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.007265    2561 manager.go:1196] Started watching for new ooms in manager
Sep 11 16:16:34.007749 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.007736    2561 manager.go:349] Starting recovery of all containers
Sep 11 16:16:34.009598 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:34.009572    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 16:16:34.009730 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:34.009604    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 16:16:34.017189 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.017154    2561 manager.go:354] Recovery completed
Sep 11 16:16:34.023107 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.023091    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:34.025263 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.025213    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:34.025263 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.025242    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:34.025263 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.025253    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:34.025862 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.025849    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 16:16:34.025862 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.025861    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 16:16:34.025977 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.025873    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:34.025977 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.025879    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 16:16:34.028094 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.028081    2561 policy_none.go:49] "None policy: Start"
Sep 11 16:16:34.028188 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.028098    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 16:16:34.028188 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.028109    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 16:16:34.028489 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.028476    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:34.028592 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.028499    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:34.028592 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.028508    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:34.030152 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.030140    2561 policy_none.go:47] "Start"
Sep 11 16:16:34.085693 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.069139    2561 manager.go:337] "Starting Device Plugin manager"
Sep 11 16:16:34.085693 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:34.069193    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 16:16:34.085693 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.069209    2561 server.go:87] "Starting device plugin registration server"
Sep 11 16:16:34.085693 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.069459    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 16:16:34.085693 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.069471    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 16:16:34.085693 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.069605    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 16:16:34.085693 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.069702    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 16:16:34.085693 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.069712    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 16:16:34.085693 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:34.070270    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 16:16:34.085693 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:34.070318    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-248.ec2.internal\" not found"
Sep 11 16:16:34.108824 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.108784    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 16:16:34.110089 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.110073    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 16:16:34.110198 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.110137    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 16:16:34.110198 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.110183    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 16:16:34.110293 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:34.110229    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 16:16:34.115783 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.115756    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:34.170019 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.169991    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:34.172422 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.172393    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:34.172422 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.172426    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:34.172557 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.172439    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:34.172557 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.172465    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-248.ec2.internal"
Sep 11 16:16:34.189217 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.189198    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-248.ec2.internal"
Sep 11 16:16:34.189274 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:34.189223    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-248.ec2.internal\": node \"ip-10-0-142-248.ec2.internal\" not found"
Sep 11 16:16:34.210305 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.210277    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-248.ec2.internal"]
Sep 11 16:16:34.210435 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.210353    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:34.214379 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.214363    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:34.214455 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.214391    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:34.214455 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.214405    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:34.216806 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.216793    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:34.216918 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.216903    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal"
Sep 11 16:16:34.216952 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.216935    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:34.217668 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.217644    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:34.217767 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.217683    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:34.217767 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.217695    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:34.217767 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.217651    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:34.217767 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.217761    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:34.217891 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.217774    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:34.218528 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:34.218512    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-248.ec2.internal\" not found"
Sep 11 16:16:34.220021 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.220008    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-248.ec2.internal"
Sep 11 16:16:34.220088 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.220034    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:34.224645 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.224630    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:34.224736 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.224657    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:34.224736 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:34.224674    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:34.234772 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:34.234754    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-248.ec2.internal\" not found" node="ip-10-0-142-248.ec2.internal"
Sep 11 16:16:34.241082 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:34.241067    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-248.ec2.internal\" not found" node="ip-10-0-142-248.ec2.internal"
Sep 11 16:16:34.318930 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:34.318857    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-248.ec2.internal\" not found"
Sep 11 16:16:35.613036 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:35.613008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bbfd3f90ca8a5cb27eebef2a7ea2cf51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal\" (UID: \"bbfd3f90ca8a5cb27eebef2a7ea2cf51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal"
Sep 11 16:16:35.613036 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:35.613040    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbfd3f90ca8a5cb27eebef2a7ea2cf51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal\" (UID: \"bbfd3f90ca8a5cb27eebef2a7ea2cf51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal"
Sep 11 16:16:35.613194 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:35.613062    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e479a5e098468c317c85809b2c11ad9e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-248.ec2.internal\" (UID: \"e479a5e098468c317c85809b2c11ad9e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-248.ec2.internal"
Sep 11 16:16:35.624099 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:35.624074    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-248.ec2.internal\" not found"
Sep 11 16:16:35.713586 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:35.713558    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bbfd3f90ca8a5cb27eebef2a7ea2cf51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal\" (UID: \"bbfd3f90ca8a5cb27eebef2a7ea2cf51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal"
Sep 11 16:16:35.713661 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:35.713522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bbfd3f90ca8a5cb27eebef2a7ea2cf51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal\" (UID: \"bbfd3f90ca8a5cb27eebef2a7ea2cf51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal"
Sep 11 16:16:35.713661 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:35.713631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbfd3f90ca8a5cb27eebef2a7ea2cf51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal\" (UID: \"bbfd3f90ca8a5cb27eebef2a7ea2cf51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal"
Sep 11 16:16:35.713661 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:35.713649    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e479a5e098468c317c85809b2c11ad9e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-248.ec2.internal\" (UID: \"e479a5e098468c317c85809b2c11ad9e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-248.ec2.internal"
Sep 11 16:16:35.713760 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:35.713671    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e479a5e098468c317c85809b2c11ad9e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-248.ec2.internal\" (UID: \"e479a5e098468c317c85809b2c11ad9e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-248.ec2.internal"
Sep 11 16:16:35.713760 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:35.713704    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbfd3f90ca8a5cb27eebef2a7ea2cf51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal\" (UID: \"bbfd3f90ca8a5cb27eebef2a7ea2cf51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal"
Sep 11 16:16:35.724669 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:35.724626    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-248.ec2.internal\" not found"
Sep 11 16:16:35.742814 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:35.742793    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal"
Sep 11 16:16:35.749837 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:35.749818    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-248.ec2.internal"
Sep 11 16:16:35.825122 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:35.825029    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-248.ec2.internal\" not found"
Sep 11 16:16:35.925626 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:35.925584    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-248.ec2.internal\" not found"
Sep 11 16:16:36.004832 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:36.004794    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 16:16:36.005348 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:36.004969    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 16:16:36.026432 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:36.026400    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-248.ec2.internal\" not found"
Sep 11 16:16:36.126598 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:36.126514    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-248.ec2.internal\" not found"
Sep 11 16:16:36.186142 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:36.186112    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:36.209959 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:36.209915    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 16:11:35 +0000 UTC" deadline="2028-06-22 12:08:04.223077596 +0000 UTC"
Sep 11 16:16:36.209959 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:36.209953    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15595h51m28.013128265s"
Sep 11 16:16:36.227559 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:36.227382    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-248.ec2.internal\" not found"
Sep 11 16:16:36.261026 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:36.260996    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:36.311995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:36.311955    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal"
Sep 11 16:16:36.347706 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:36.347679    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 16:16:36.348605 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:36.348591    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-248.ec2.internal"
Sep 11 16:16:36.369451 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:36.369431    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 16:16:36.373472 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:36.373436    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode479a5e098468c317c85809b2c11ad9e.slice/crio-a6e82ebd7b006a0e6c490798025438af6ab33812d5e29fb6bfb4be941ad05219 WatchSource:0}: Error finding container a6e82ebd7b006a0e6c490798025438af6ab33812d5e29fb6bfb4be941ad05219: Status 404 returned error can't find the container with id a6e82ebd7b006a0e6c490798025438af6ab33812d5e29fb6bfb4be941ad05219
Sep 11 16:16:36.373926 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:36.373904    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfd3f90ca8a5cb27eebef2a7ea2cf51.slice/crio-a36472b509a87ed62762372b7dea69551b7edd81b04e6efa9d0e727efabed37b WatchSource:0}: Error finding container a36472b509a87ed62762372b7dea69551b7edd81b04e6efa9d0e727efabed37b: Status 404 returned error can't find the container with id a36472b509a87ed62762372b7dea69551b7edd81b04e6efa9d0e727efabed37b
Sep 11 16:16:36.378543 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:36.378524    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:16:36.434810 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:36.434784    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:37.052633 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.052607    2561 apiserver.go:52] "Watching apiserver"
Sep 11 16:16:37.059713 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.059677    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 16:16:37.060610 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.060583    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w","openshift-cluster-node-tuning-operator/tuned-9mpsx","openshift-multus/multus-5jvnz","openshift-multus/multus-additional-cni-plugins-7xmqw","openshift-multus/network-metrics-daemon-z6xtq","kube-system/global-pull-secret-syncer-xlpvp","kube-system/kube-apiserver-proxy-ip-10-0-142-248.ec2.internal","openshift-dns/node-resolver-txkjh","openshift-image-registry/node-ca-2lrtq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal","openshift-network-diagnostics/network-check-target-qdmr8","openshift-network-operator/iptables-alerter-cnprz","openshift-ovn-kubernetes/ovnkube-node-sz5sg","kube-system/konnectivity-agent-xwsg5"]
Sep 11 16:16:37.063341 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.063315    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.065804 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.065786    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.067816 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.067789    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.070301 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.070279    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.072708 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.072510    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:37.072708 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.072581    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:16:37.075156 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.075133    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 16:16:37.075260 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.075162    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:37.075260 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.075247    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:16:37.075377 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.075280    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 16:16:37.075377 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.075368    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 16:16:37.075471 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.075427    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 16:16:37.075471 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.075441    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 16:16:37.075471 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.075460    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 16:16:37.075608 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.075524    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 16:16:37.075608 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.075542    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 16:16:37.075698 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.075615    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 16:16:37.075698 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.075633    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 16:16:37.075698 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.075695    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 16:16:37.075834 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.075703    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-l224q\""
Sep 11 16:16:37.075834 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.075709    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 16:16:37.075834 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.075779    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:16:37.075989 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.075965    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g2qss\""
Sep 11 16:16:37.076128 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.076091    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g8sbm\""
Sep 11 16:16:37.076128 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.076093    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vlqpv\""
Sep 11 16:16:37.077358 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.077338    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.081429 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.081408    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 16:16:37.082680 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.082658    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-txkjh"
Sep 11 16:16:37.083553 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.083533    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 16:16:37.083735 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.083718    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5w64d\""
Sep 11 16:16:37.083982 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.083965    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 16:16:37.084073 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.084003    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 16:16:37.085630 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.085609    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-db5rw\""
Sep 11 16:16:37.085753 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.085737    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 16:16:37.086558 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.086538    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 16:16:37.086877 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.086849    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2lrtq"
Sep 11 16:16:37.088826 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.088806    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xwsg5"
Sep 11 16:16:37.089019 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.088999    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6mnjx\""
Sep 11 16:16:37.089105 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.088917    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 16:16:37.089162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.089106    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 16:16:37.089503 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.089485    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 16:16:37.091464 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.091264    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:37.091464 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.091333    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:16:37.092112 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.092094    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fjhqj\""
Sep 11 16:16:37.093620 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.093600    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 16:16:37.093757 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.093600    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 16:16:37.093838 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.093821    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cnprz"
Sep 11 16:16:37.095963 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.095940    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 16:16:37.096246 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.096226    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 16:16:37.096457 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.096441    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-48z7c\""
Sep 11 16:16:37.096720 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.096703    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:16:37.113744 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.113721    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 16:16:37.119345 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-var-lib-kubelet\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.119474 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119343    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.119474 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119386    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-multus-socket-dir-parent\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.119474 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119410    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-sysconfig\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.119474 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119432    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-lib-modules\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.119474 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119452    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-var-lib-kubelet\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.119769 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119495    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-multus-conf-dir\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.119769 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119531    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw44k\" (UniqueName: \"kubernetes.io/projected/13f82d90-a9ab-4751-affc-6fef32b07c68-kube-api-access-kw44k\") pod \"iptables-alerter-cnprz\" (UID: \"13f82d90-a9ab-4751-affc-6fef32b07c68\") " pod="openshift-network-operator/iptables-alerter-cnprz"
Sep 11 16:16:37.119769 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-cni-netd\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.119769 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-modprobe-d\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.119769 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119713    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f76398de-0983-4cee-afab-0e48e5b8db38-host\") pod \"node-ca-2lrtq\" (UID: \"f76398de-0983-4cee-afab-0e48e5b8db38\") " pod="openshift-image-registry/node-ca-2lrtq"
Sep 11 16:16:37.119769 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119750    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-hostroot\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.120129 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119795    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret\") pod \"global-pull-secret-syncer-xlpvp\" (UID: \"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f\") " pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:37.120129 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119825    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-sysctl-d\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.120129 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119850    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98x8m\" (UniqueName: \"kubernetes.io/projected/b15c0e16-1077-454c-aafa-16902113243c-kube-api-access-98x8m\") pod \"network-metrics-daemon-z6xtq\" (UID: \"b15c0e16-1077-454c-aafa-16902113243c\") " pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:37.120129 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119879    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e86cc7cc-d70a-428d-abc9-4e68716d13d0-konnectivity-ca\") pod \"konnectivity-agent-xwsg5\" (UID: \"e86cc7cc-d70a-428d-abc9-4e68716d13d0\") " pod="kube-system/konnectivity-agent-xwsg5"
Sep 11 16:16:37.120129 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119904    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-kubelet\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.120129 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-run-netns\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.120129 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119936    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-sys\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.120129 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.119968    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1774723c-dc46-4bdf-b1a9-dc3de662c556-tmp\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.120129 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120013    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-socket-dir\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.120129 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kvw2\" (UniqueName: \"kubernetes.io/projected/062d3c49-c0e6-400d-bf42-5ef08d0c83d7-kube-api-access-9kvw2\") pod \"node-resolver-txkjh\" (UID: \"062d3c49-c0e6-400d-bf42-5ef08d0c83d7\") " pod="openshift-dns/node-resolver-txkjh"
Sep 11 16:16:37.120129 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-ovnkube-script-lib\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.120662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120137    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-sysctl-conf\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.120662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120187    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-host\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.120662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062d3c49-c0e6-400d-bf42-5ef08d0c83d7-tmp-dir\") pod \"node-resolver-txkjh\" (UID: \"062d3c49-c0e6-400d-bf42-5ef08d0c83d7\") " pod="openshift-dns/node-resolver-txkjh"
Sep 11 16:16:37.120662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-env-overrides\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.120662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120289    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-systemd\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.120662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120338    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spv28\" (UniqueName: \"kubernetes.io/projected/f76398de-0983-4cee-afab-0e48e5b8db38-kube-api-access-spv28\") pod \"node-ca-2lrtq\" (UID: \"f76398de-0983-4cee-afab-0e48e5b8db38\") " pod="openshift-image-registry/node-ca-2lrtq"
Sep 11 16:16:37.120662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120361    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-run-multus-certs\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.120662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120398    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53bba805-9236-4e17-bcb2-a4b29b4a2355-cnibin\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.120662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120430    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6rfb\" (UniqueName: \"kubernetes.io/projected/53bba805-9236-4e17-bcb2-a4b29b4a2355-kube-api-access-b6rfb\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.120662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120499    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-sys-fs\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.120662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120531    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53bba805-9236-4e17-bcb2-a4b29b4a2355-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.120662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120563    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-dbus\") pod \"global-pull-secret-syncer-xlpvp\" (UID: \"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f\") " pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:37.120662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120587    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzp7f\" (UniqueName: \"kubernetes.io/projected/1774723c-dc46-4bdf-b1a9-dc3de662c556-kube-api-access-bzp7f\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.120662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-cni-binary-copy\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.120662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-systemd-units\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.121343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120690    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-var-lib-openvswitch\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.121343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120785    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-run-ovn-kubernetes\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.121343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-registration-dir\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.121343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-device-dir\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.121343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120901    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/13f82d90-a9ab-4751-affc-6fef32b07c68-iptables-alerter-script\") pod \"iptables-alerter-cnprz\" (UID: \"13f82d90-a9ab-4751-affc-6fef32b07c68\") " pod="openshift-network-operator/iptables-alerter-cnprz"
Sep 11 16:16:37.121343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120926    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-run-openvswitch\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.121343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120950    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-run-ovn\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.121343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.120974    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.121343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-kubernetes\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.121343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121040    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-tuned\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.121343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs\") pod \"network-metrics-daemon-z6xtq\" (UID: \"b15c0e16-1077-454c-aafa-16902113243c\") " pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:37.121343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-cnibin\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.121343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121139    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-run-netns\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.121343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121202    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-slash\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.121343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53bba805-9236-4e17-bcb2-a4b29b4a2355-cni-binary-copy\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.121343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-kubelet-config\") pod \"global-pull-secret-syncer-xlpvp\" (UID: \"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f\") " pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:37.121995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbndv\" (UniqueName: \"kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv\") pod \"network-check-target-qdmr8\" (UID: \"af5da358-605f-4f1f-a20a-5aa63fb8889d\") " pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:37.121995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e86cc7cc-d70a-428d-abc9-4e68716d13d0-agent-certs\") pod \"konnectivity-agent-xwsg5\" (UID: \"e86cc7cc-d70a-428d-abc9-4e68716d13d0\") " pod="kube-system/konnectivity-agent-xwsg5"
Sep 11 16:16:37.121995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121337    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-etc-openvswitch\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.121995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121359    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-cni-bin\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.121995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121404    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-ovnkube-config\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.121995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121437    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5p5b\" (UniqueName: \"kubernetes.io/projected/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-kube-api-access-j5p5b\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.121995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121517    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/062d3c49-c0e6-400d-bf42-5ef08d0c83d7-hosts-file\") pod \"node-resolver-txkjh\" (UID: \"062d3c49-c0e6-400d-bf42-5ef08d0c83d7\") " pod="openshift-dns/node-resolver-txkjh"
Sep 11 16:16:37.121995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121553    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-etc-kubernetes\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.121995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13f82d90-a9ab-4751-affc-6fef32b07c68-host-slash\") pod \"iptables-alerter-cnprz\" (UID: \"13f82d90-a9ab-4751-affc-6fef32b07c68\") " pod="openshift-network-operator/iptables-alerter-cnprz"
Sep 11 16:16:37.121995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121597    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f76398de-0983-4cee-afab-0e48e5b8db38-serviceca\") pod \"node-ca-2lrtq\" (UID: \"f76398de-0983-4cee-afab-0e48e5b8db38\") " pod="openshift-image-registry/node-ca-2lrtq"
Sep 11 16:16:37.121995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121619    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-system-cni-dir\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.121995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121642    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsdm5\" (UniqueName: \"kubernetes.io/projected/d85a376a-563f-4ce7-b0de-6bf847191c83-kube-api-access-gsdm5\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.121995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121673    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-run-k8s-cni-cncf-io\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.121995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121695    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-var-lib-cni-bin\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.121995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121717    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-var-lib-cni-multus\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.121995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-multus-daemon-config\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.122739 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121761    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-run-systemd\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.122739 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-ovn-node-metrics-cert\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.122739 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-etc-selinux\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.122739 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121831    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8fd4\" (UniqueName: \"kubernetes.io/projected/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-kube-api-access-d8fd4\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.122739 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121884    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53bba805-9236-4e17-bcb2-a4b29b4a2355-system-cni-dir\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.122739 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121905    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53bba805-9236-4e17-bcb2-a4b29b4a2355-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.122739 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121930    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53bba805-9236-4e17-bcb2-a4b29b4a2355-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.122739 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121955    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-log-socket\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.122739 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121974    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-multus-cni-dir\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.122739 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.121994    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-node-log\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.122739 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.122014    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-run\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.211186 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.211115    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 16:16:37.222698 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.222669    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-log-socket\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.222698 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.222705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-multus-cni-dir\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.222933 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.222728    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-node-log\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.222933 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.222782    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-node-log\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.222933 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.222789    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-log-socket\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.222933 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.222818    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-run\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.222933 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.222840    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-var-lib-kubelet\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.222933 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.222860    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.222933 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.222883    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-multus-socket-dir-parent\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.222933 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.222918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-sysconfig\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.222942    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-run\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.222961    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-lib-modules\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.222939    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-var-lib-kubelet\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.222948    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-multus-cni-dir\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.222991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-var-lib-kubelet\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223006    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-multus-socket-dir-parent\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-multus-conf-dir\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223022    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-sysconfig\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223060    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-var-lib-kubelet\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223068    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-lib-modules\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223075    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223076    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-multus-conf-dir\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223091    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw44k\" (UniqueName: \"kubernetes.io/projected/13f82d90-a9ab-4751-affc-6fef32b07c68-kube-api-access-kw44k\") pod \"iptables-alerter-cnprz\" (UID: \"13f82d90-a9ab-4751-affc-6fef32b07c68\") " pod="openshift-network-operator/iptables-alerter-cnprz"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-cni-netd\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223135    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-modprobe-d\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f76398de-0983-4cee-afab-0e48e5b8db38-host\") pod \"node-ca-2lrtq\" (UID: \"f76398de-0983-4cee-afab-0e48e5b8db38\") " pod="openshift-image-registry/node-ca-2lrtq"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f76398de-0983-4cee-afab-0e48e5b8db38-host\") pod \"node-ca-2lrtq\" (UID: \"f76398de-0983-4cee-afab-0e48e5b8db38\") " pod="openshift-image-registry/node-ca-2lrtq"
Sep 11 16:16:37.223327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223226    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-cni-netd\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.224015 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-hostroot\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.224015 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223302    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret\") pod \"global-pull-secret-syncer-xlpvp\" (UID: \"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f\") " pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:37.224015 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223314    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-hostroot\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.224015 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223317    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-sysctl-d\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.224015 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98x8m\" (UniqueName: \"kubernetes.io/projected/b15c0e16-1077-454c-aafa-16902113243c-kube-api-access-98x8m\") pod \"network-metrics-daemon-z6xtq\" (UID: \"b15c0e16-1077-454c-aafa-16902113243c\") " pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:37.224015 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223392    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-sysctl-d\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.224015 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.223396    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:37.224015 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223272    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-modprobe-d\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.224015 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223396    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e86cc7cc-d70a-428d-abc9-4e68716d13d0-konnectivity-ca\") pod \"konnectivity-agent-xwsg5\" (UID: \"e86cc7cc-d70a-428d-abc9-4e68716d13d0\") " pod="kube-system/konnectivity-agent-xwsg5"
Sep 11 16:16:37.224015 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.223490    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret podName:b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f nodeName:}" failed. No retries permitted until 2026-09-11 16:16:37.723453824 +0000 UTC m=+3.063547862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret") pod "global-pull-secret-syncer-xlpvp" (UID: "b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:37.224015 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-kubelet\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.224015 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223544    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-run-netns\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.224015 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.223566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-sys\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.224677 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.224295    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-kubelet\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.224677 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.224321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-run-netns\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.224677 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.224366    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1774723c-dc46-4bdf-b1a9-dc3de662c556-tmp\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.224677 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.224406    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-sys\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.224677 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.224414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-socket-dir\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.224677 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.224459    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kvw2\" (UniqueName: \"kubernetes.io/projected/062d3c49-c0e6-400d-bf42-5ef08d0c83d7-kube-api-access-9kvw2\") pod \"node-resolver-txkjh\" (UID: \"062d3c49-c0e6-400d-bf42-5ef08d0c83d7\") " pod="openshift-dns/node-resolver-txkjh"
Sep 11 16:16:37.224677 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.224491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-ovnkube-script-lib\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.224677 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.224523    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-sysctl-conf\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.224677 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.224547    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-host\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.224677 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.224576    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062d3c49-c0e6-400d-bf42-5ef08d0c83d7-tmp-dir\") pod \"node-resolver-txkjh\" (UID: \"062d3c49-c0e6-400d-bf42-5ef08d0c83d7\") " pod="openshift-dns/node-resolver-txkjh"
Sep 11 16:16:37.225139 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.224761    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-sysctl-conf\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.225139 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.224779    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e86cc7cc-d70a-428d-abc9-4e68716d13d0-konnectivity-ca\") pod \"konnectivity-agent-xwsg5\" (UID: \"e86cc7cc-d70a-428d-abc9-4e68716d13d0\") " pod="kube-system/konnectivity-agent-xwsg5"
Sep 11 16:16:37.225139 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.224863    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-socket-dir\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.225139 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.224890    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062d3c49-c0e6-400d-bf42-5ef08d0c83d7-tmp-dir\") pod \"node-resolver-txkjh\" (UID: \"062d3c49-c0e6-400d-bf42-5ef08d0c83d7\") " pod="openshift-dns/node-resolver-txkjh"
Sep 11 16:16:37.225139 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.224894    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-env-overrides\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.225139 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.224982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-systemd\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.225139 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spv28\" (UniqueName: \"kubernetes.io/projected/f76398de-0983-4cee-afab-0e48e5b8db38-kube-api-access-spv28\") pod \"node-ca-2lrtq\" (UID: \"f76398de-0983-4cee-afab-0e48e5b8db38\") " pod="openshift-image-registry/node-ca-2lrtq"
Sep 11 16:16:37.225139 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225043    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-run-multus-certs\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.225139 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53bba805-9236-4e17-bcb2-a4b29b4a2355-cnibin\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.225139 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225128    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-host\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.225642 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225131    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53bba805-9236-4e17-bcb2-a4b29b4a2355-cnibin\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.225642 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-systemd\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.225642 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225204    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6rfb\" (UniqueName: \"kubernetes.io/projected/53bba805-9236-4e17-bcb2-a4b29b4a2355-kube-api-access-b6rfb\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.225642 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-ovnkube-script-lib\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.225642 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225289    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-run-multus-certs\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.225642 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225344    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-env-overrides\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.225642 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-sys-fs\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.225642 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53bba805-9236-4e17-bcb2-a4b29b4a2355-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.225642 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225530    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-dbus\") pod \"global-pull-secret-syncer-xlpvp\" (UID: \"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f\") " pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225620    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225657    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-sys-fs\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225694    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53bba805-9236-4e17-bcb2-a4b29b4a2355-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzp7f\" (UniqueName: \"kubernetes.io/projected/1774723c-dc46-4bdf-b1a9-dc3de662c556-kube-api-access-bzp7f\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-cni-binary-copy\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225700    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-dbus\") pod \"global-pull-secret-syncer-xlpvp\" (UID: \"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f\") " pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-systemd-units\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-var-lib-openvswitch\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225872    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-run-ovn-kubernetes\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225899    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-registration-dir\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225932    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-device-dir\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.225981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/13f82d90-a9ab-4751-affc-6fef32b07c68-iptables-alerter-script\") pod \"iptables-alerter-cnprz\" (UID: \"13f82d90-a9ab-4751-affc-6fef32b07c68\") " pod="openshift-network-operator/iptables-alerter-cnprz"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226011    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-run-ovn-kubernetes\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226022    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-run-openvswitch\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226051    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-run-ovn\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226079    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-var-lib-openvswitch\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.226113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226091    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226327    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-run-ovn\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226353    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-cni-binary-copy\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226401    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-registration-dir\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226468    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-kubernetes\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226484    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-device-dir\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-tuned\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226553    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-systemd-units\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226562    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-run-openvswitch\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs\") pod \"network-metrics-daemon-z6xtq\" (UID: \"b15c0e16-1077-454c-aafa-16902113243c\") " pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226626    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-cnibin\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226642    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-kubernetes\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-run-netns\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-slash\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226711    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53bba805-9236-4e17-bcb2-a4b29b4a2355-cni-binary-copy\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226716    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/13f82d90-a9ab-4751-affc-6fef32b07c68-iptables-alerter-script\") pod \"iptables-alerter-cnprz\" (UID: \"13f82d90-a9ab-4751-affc-6fef32b07c68\") " pod="openshift-network-operator/iptables-alerter-cnprz"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226766    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-run-netns\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226768    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-cnibin\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.226903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226795    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-kubelet-config\") pod \"global-pull-secret-syncer-xlpvp\" (UID: \"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f\") " pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-slash\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226833    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbndv\" (UniqueName: \"kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv\") pod \"network-check-target-qdmr8\" (UID: \"af5da358-605f-4f1f-a20a-5aa63fb8889d\") " pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.226855    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e86cc7cc-d70a-428d-abc9-4e68716d13d0-agent-certs\") pod \"konnectivity-agent-xwsg5\" (UID: \"e86cc7cc-d70a-428d-abc9-4e68716d13d0\") " pod="kube-system/konnectivity-agent-xwsg5"
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226894    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-etc-openvswitch\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.226915    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs podName:b15c0e16-1077-454c-aafa-16902113243c nodeName:}" failed. No retries permitted until 2026-09-11 16:16:37.726891825 +0000 UTC m=+3.066985850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs") pod "network-metrics-daemon-z6xtq" (UID: "b15c0e16-1077-454c-aafa-16902113243c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226939    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-etc-openvswitch\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-cni-bin\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.226992    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-cni-bin\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227002    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-ovnkube-config\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227043    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5p5b\" (UniqueName: \"kubernetes.io/projected/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-kube-api-access-j5p5b\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227071    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/062d3c49-c0e6-400d-bf42-5ef08d0c83d7-hosts-file\") pod \"node-resolver-txkjh\" (UID: \"062d3c49-c0e6-400d-bf42-5ef08d0c83d7\") " pod="openshift-dns/node-resolver-txkjh"
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-etc-kubernetes\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13f82d90-a9ab-4751-affc-6fef32b07c68-host-slash\") pod \"iptables-alerter-cnprz\" (UID: \"13f82d90-a9ab-4751-affc-6fef32b07c68\") " pod="openshift-network-operator/iptables-alerter-cnprz"
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227161    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f76398de-0983-4cee-afab-0e48e5b8db38-serviceca\") pod \"node-ca-2lrtq\" (UID: \"f76398de-0983-4cee-afab-0e48e5b8db38\") " pod="openshift-image-registry/node-ca-2lrtq"
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-system-cni-dir\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.227741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsdm5\" (UniqueName: \"kubernetes.io/projected/d85a376a-563f-4ce7-b0de-6bf847191c83-kube-api-access-gsdm5\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.228541 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-run-k8s-cni-cncf-io\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.228541 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227311    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-var-lib-cni-bin\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.228541 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227343    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-var-lib-cni-multus\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.228541 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-multus-daemon-config\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.228541 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227399    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53bba805-9236-4e17-bcb2-a4b29b4a2355-cni-binary-copy\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.228541 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-ovnkube-config\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.228541 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/062d3c49-c0e6-400d-bf42-5ef08d0c83d7-hosts-file\") pod \"node-resolver-txkjh\" (UID: \"062d3c49-c0e6-400d-bf42-5ef08d0c83d7\") " pod="openshift-dns/node-resolver-txkjh"
Sep 11 16:16:37.228541 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227560    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-etc-kubernetes\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.228541 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227610    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.228541 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227696    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-var-lib-cni-multus\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.228541 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227702    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-kubelet-config\") pod \"global-pull-secret-syncer-xlpvp\" (UID: \"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f\") " pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:37.228541 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227960    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13f82d90-a9ab-4751-affc-6fef32b07c68-host-slash\") pod \"iptables-alerter-cnprz\" (UID: \"13f82d90-a9ab-4751-affc-6fef32b07c68\") " pod="openshift-network-operator/iptables-alerter-cnprz"
Sep 11 16:16:37.228541 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.227977    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-multus-daemon-config\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.228541 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.228043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-system-cni-dir\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.228541 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.228088    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-var-lib-cni-bin\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.228541 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.228128    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-host-run-k8s-cni-cncf-io\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.229288 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.228743    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f76398de-0983-4cee-afab-0e48e5b8db38-serviceca\") pod \"node-ca-2lrtq\" (UID: \"f76398de-0983-4cee-afab-0e48e5b8db38\") " pod="openshift-image-registry/node-ca-2lrtq"
Sep 11 16:16:37.229288 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.228810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-run-systemd\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.229288 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.228855    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-ovn-node-metrics-cert\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.229288 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.228889    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-etc-selinux\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.229288 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.228916    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8fd4\" (UniqueName: \"kubernetes.io/projected/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-kube-api-access-d8fd4\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.229508 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.229412    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53bba805-9236-4e17-bcb2-a4b29b4a2355-system-cni-dir\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.229508 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.229448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53bba805-9236-4e17-bcb2-a4b29b4a2355-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.229508 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.229493    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53bba805-9236-4e17-bcb2-a4b29b4a2355-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.230305 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.229977    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-run-systemd\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.230305 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.230096    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53bba805-9236-4e17-bcb2-a4b29b4a2355-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.230305 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.230196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53bba805-9236-4e17-bcb2-a4b29b4a2355-system-cni-dir\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.230305 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.230270    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d85a376a-563f-4ce7-b0de-6bf847191c83-etc-selinux\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.230695 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.230669    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53bba805-9236-4e17-bcb2-a4b29b4a2355-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.230839 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.230787    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1774723c-dc46-4bdf-b1a9-dc3de662c556-etc-tuned\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.233164 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.232796    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1774723c-dc46-4bdf-b1a9-dc3de662c556-tmp\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.233164 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.233102    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e86cc7cc-d70a-428d-abc9-4e68716d13d0-agent-certs\") pod \"konnectivity-agent-xwsg5\" (UID: \"e86cc7cc-d70a-428d-abc9-4e68716d13d0\") " pod="kube-system/konnectivity-agent-xwsg5"
Sep 11 16:16:37.233575 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.233534    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-ovn-node-metrics-cert\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.237385 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.236995    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 16:16:37.237385 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.237375    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98x8m\" (UniqueName: \"kubernetes.io/projected/b15c0e16-1077-454c-aafa-16902113243c-kube-api-access-98x8m\") pod \"network-metrics-daemon-z6xtq\" (UID: \"b15c0e16-1077-454c-aafa-16902113243c\") " pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:37.237542 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.237477    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw44k\" (UniqueName: \"kubernetes.io/projected/13f82d90-a9ab-4751-affc-6fef32b07c68-kube-api-access-kw44k\") pod \"iptables-alerter-cnprz\" (UID: \"13f82d90-a9ab-4751-affc-6fef32b07c68\") " pod="openshift-network-operator/iptables-alerter-cnprz"
Sep 11 16:16:37.239537 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.239513    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kvw2\" (UniqueName: \"kubernetes.io/projected/062d3c49-c0e6-400d-bf42-5ef08d0c83d7-kube-api-access-9kvw2\") pod \"node-resolver-txkjh\" (UID: \"062d3c49-c0e6-400d-bf42-5ef08d0c83d7\") " pod="openshift-dns/node-resolver-txkjh"
Sep 11 16:16:37.243062 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.243028    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6rfb\" (UniqueName: \"kubernetes.io/projected/53bba805-9236-4e17-bcb2-a4b29b4a2355-kube-api-access-b6rfb\") pod \"multus-additional-cni-plugins-7xmqw\" (UID: \"53bba805-9236-4e17-bcb2-a4b29b4a2355\") " pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.245154 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.245135    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spv28\" (UniqueName: \"kubernetes.io/projected/f76398de-0983-4cee-afab-0e48e5b8db38-kube-api-access-spv28\") pod \"node-ca-2lrtq\" (UID: \"f76398de-0983-4cee-afab-0e48e5b8db38\") " pod="openshift-image-registry/node-ca-2lrtq"
Sep 11 16:16:37.246042 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.246014    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzp7f\" (UniqueName: \"kubernetes.io/projected/1774723c-dc46-4bdf-b1a9-dc3de662c556-kube-api-access-bzp7f\") pod \"tuned-9mpsx\" (UID: \"1774723c-dc46-4bdf-b1a9-dc3de662c556\") " pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.246951 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.246931    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:16:37.247040 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.246955    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:16:37.247040 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.246969    2561 projected.go:196] Error preparing data for projected volume kube-api-access-mbndv for pod openshift-network-diagnostics/network-check-target-qdmr8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:37.247040 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.247039    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv podName:af5da358-605f-4f1f-a20a-5aa63fb8889d nodeName:}" failed. No retries permitted until 2026-09-11 16:16:37.747023584 +0000 UTC m=+3.087117608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mbndv" (UniqueName: "kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv") pod "network-check-target-qdmr8" (UID: "af5da358-605f-4f1f-a20a-5aa63fb8889d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:37.250145 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.250121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsdm5\" (UniqueName: \"kubernetes.io/projected/d85a376a-563f-4ce7-b0de-6bf847191c83-kube-api-access-gsdm5\") pod \"aws-ebs-csi-driver-node-glc8w\" (UID: \"d85a376a-563f-4ce7-b0de-6bf847191c83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.251807 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.251780    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8fd4\" (UniqueName: \"kubernetes.io/projected/346d7ed3-37a8-43fd-8544-96f9bc2fb32f-kube-api-access-d8fd4\") pod \"multus-5jvnz\" (UID: \"346d7ed3-37a8-43fd-8544-96f9bc2fb32f\") " pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.252306 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.252286    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5p5b\" (UniqueName: \"kubernetes.io/projected/38d0a297-96bb-437e-ac8d-7ef56f3c3c87-kube-api-access-j5p5b\") pod \"ovnkube-node-sz5sg\" (UID: \"38d0a297-96bb-437e-ac8d-7ef56f3c3c87\") " pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.257414 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.257389    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jgc5h"
Sep 11 16:16:37.267224 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.267200    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jgc5h"
Sep 11 16:16:37.319596 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.319490    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-248.ec2.internal" event={"ID":"e479a5e098468c317c85809b2c11ad9e","Type":"ContainerStarted","Data":"a6e82ebd7b006a0e6c490798025438af6ab33812d5e29fb6bfb4be941ad05219"}
Sep 11 16:16:37.320677 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.320642    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal" event={"ID":"bbfd3f90ca8a5cb27eebef2a7ea2cf51","Type":"ContainerStarted","Data":"a36472b509a87ed62762372b7dea69551b7edd81b04e6efa9d0e727efabed37b"}
Sep 11 16:16:37.377389 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.377350    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:16:37.384544 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.384519    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9mpsx"
Sep 11 16:16:37.396453 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.396426    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5jvnz"
Sep 11 16:16:37.403239 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.403219    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7xmqw"
Sep 11 16:16:37.411250 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.411221    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w"
Sep 11 16:16:37.419234 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.419206    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-txkjh"
Sep 11 16:16:37.427102 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.427080    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2lrtq"
Sep 11 16:16:37.434836 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.434816    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xwsg5"
Sep 11 16:16:37.441466 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.441446    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cnprz"
Sep 11 16:16:37.569138 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.569106    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:37.732988 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.732953    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret\") pod \"global-pull-secret-syncer-xlpvp\" (UID: \"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f\") " pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:37.733199 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.733006    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs\") pod \"network-metrics-daemon-z6xtq\" (UID: \"b15c0e16-1077-454c-aafa-16902113243c\") " pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:37.733199 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.733110    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:37.733199 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.733168    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs podName:b15c0e16-1077-454c-aafa-16902113243c nodeName:}" failed. No retries permitted until 2026-09-11 16:16:38.733149181 +0000 UTC m=+4.073243207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs") pod "network-metrics-daemon-z6xtq" (UID: "b15c0e16-1077-454c-aafa-16902113243c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:37.733199 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.733111    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:37.733385 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.733252    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret podName:b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f nodeName:}" failed. No retries permitted until 2026-09-11 16:16:38.73323717 +0000 UTC m=+4.073331216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret") pod "global-pull-secret-syncer-xlpvp" (UID: "b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:37.834190 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:37.834136    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbndv\" (UniqueName: \"kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv\") pod \"network-check-target-qdmr8\" (UID: \"af5da358-605f-4f1f-a20a-5aa63fb8889d\") " pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:37.834370 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.834320    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:16:37.834370 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.834351    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:16:37.834370 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.834365    2561 projected.go:196] Error preparing data for projected volume kube-api-access-mbndv for pod openshift-network-diagnostics/network-check-target-qdmr8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:37.834513 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:37.834431    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv podName:af5da358-605f-4f1f-a20a-5aa63fb8889d nodeName:}" failed. No retries permitted until 2026-09-11 16:16:38.834414762 +0000 UTC m=+4.174508794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbndv" (UniqueName: "kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv") pod "network-check-target-qdmr8" (UID: "af5da358-605f-4f1f-a20a-5aa63fb8889d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:38.256126 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:38.256092    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d0a297_96bb_437e_ac8d_7ef56f3c3c87.slice/crio-b55cd9d100ac937abdd62b02011633db565b0471bab2e4dcc17d2a5694ee59a3 WatchSource:0}: Error finding container b55cd9d100ac937abdd62b02011633db565b0471bab2e4dcc17d2a5694ee59a3: Status 404 returned error can't find the container with id b55cd9d100ac937abdd62b02011633db565b0471bab2e4dcc17d2a5694ee59a3
Sep 11 16:16:38.258024 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:38.257987    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85a376a_563f_4ce7_b0de_6bf847191c83.slice/crio-309b45dbc3876f29f18f9ed021b6074bcd9577fd7879b56866a1a3b407a43269 WatchSource:0}: Error finding container 309b45dbc3876f29f18f9ed021b6074bcd9577fd7879b56866a1a3b407a43269: Status 404 returned error can't find the container with id 309b45dbc3876f29f18f9ed021b6074bcd9577fd7879b56866a1a3b407a43269
Sep 11 16:16:38.259244 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:38.259130    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bba805_9236_4e17_bcb2_a4b29b4a2355.slice/crio-1b4718432b38052f624e22093dd4bd7da992f92d07809ba30176ad623c97dfcd WatchSource:0}: Error finding container 1b4718432b38052f624e22093dd4bd7da992f92d07809ba30176ad623c97dfcd: Status 404 returned error can't find the container with id 1b4718432b38052f624e22093dd4bd7da992f92d07809ba30176ad623c97dfcd
Sep 11 16:16:38.263163 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:38.263116    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1774723c_dc46_4bdf_b1a9_dc3de662c556.slice/crio-555b26c82db26b408de0082dfcba4aa9519ec741e88f1fd3a361245912d2e519 WatchSource:0}: Error finding container 555b26c82db26b408de0082dfcba4aa9519ec741e88f1fd3a361245912d2e519: Status 404 returned error can't find the container with id 555b26c82db26b408de0082dfcba4aa9519ec741e88f1fd3a361245912d2e519
Sep 11 16:16:38.264614 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:38.264549    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f82d90_a9ab_4751_affc_6fef32b07c68.slice/crio-28125660c656249d3470e3359344f61fc2cdd40939646beb99a207c1e6691bfa WatchSource:0}: Error finding container 28125660c656249d3470e3359344f61fc2cdd40939646beb99a207c1e6691bfa: Status 404 returned error can't find the container with id 28125660c656249d3470e3359344f61fc2cdd40939646beb99a207c1e6691bfa
Sep 11 16:16:38.265524 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:16:38.265472    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod062d3c49_c0e6_400d_bf42_5ef08d0c83d7.slice/crio-0b70bcfc05ab276d90d2c29c728dde9c5416f6c7ecf25a8352d5972645c0d444 WatchSource:0}: Error finding container 0b70bcfc05ab276d90d2c29c728dde9c5416f6c7ecf25a8352d5972645c0d444: Status 404 returned error can't find the container with id 0b70bcfc05ab276d90d2c29c728dde9c5416f6c7ecf25a8352d5972645c0d444
Sep 11 16:16:38.271307 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:38.271275    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 16:11:37 +0000 UTC" deadline="2028-04-14 18:10:19.756841531 +0000 UTC"
Sep 11 16:16:38.271307 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:38.271302    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13945h53m41.485544323s"
Sep 11 16:16:38.315506 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:38.315478    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:38.315611 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:38.315582    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:16:38.324724 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:38.324681    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cnprz" event={"ID":"13f82d90-a9ab-4751-affc-6fef32b07c68","Type":"ContainerStarted","Data":"28125660c656249d3470e3359344f61fc2cdd40939646beb99a207c1e6691bfa"}
Sep 11 16:16:38.325894 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:38.325865    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xmqw" event={"ID":"53bba805-9236-4e17-bcb2-a4b29b4a2355","Type":"ContainerStarted","Data":"1b4718432b38052f624e22093dd4bd7da992f92d07809ba30176ad623c97dfcd"}
Sep 11 16:16:38.327225 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:38.327162    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w" event={"ID":"d85a376a-563f-4ce7-b0de-6bf847191c83","Type":"ContainerStarted","Data":"309b45dbc3876f29f18f9ed021b6074bcd9577fd7879b56866a1a3b407a43269"}
Sep 11 16:16:38.328358 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:38.328337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg" event={"ID":"38d0a297-96bb-437e-ac8d-7ef56f3c3c87","Type":"ContainerStarted","Data":"b55cd9d100ac937abdd62b02011633db565b0471bab2e4dcc17d2a5694ee59a3"}
Sep 11 16:16:38.329510 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:38.329488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5jvnz" event={"ID":"346d7ed3-37a8-43fd-8544-96f9bc2fb32f","Type":"ContainerStarted","Data":"ce43f4f74f76c9dd27d6c8337e857c69f1e13aace7debe9d581e0cb54aa7edf4"}
Sep 11 16:16:38.330564 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:38.330539    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xwsg5" event={"ID":"e86cc7cc-d70a-428d-abc9-4e68716d13d0","Type":"ContainerStarted","Data":"39bf2a15e4612e51863d49c71dfd945c1eb7d724c36818626f3074764d979b2b"}
Sep 11 16:16:38.331599 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:38.331574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2lrtq" event={"ID":"f76398de-0983-4cee-afab-0e48e5b8db38","Type":"ContainerStarted","Data":"dd2dc6193e14eea5bc5792412694eb304361534a2b809891a4c0ee790b14a374"}
Sep 11 16:16:38.332597 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:38.332574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-txkjh" event={"ID":"062d3c49-c0e6-400d-bf42-5ef08d0c83d7","Type":"ContainerStarted","Data":"0b70bcfc05ab276d90d2c29c728dde9c5416f6c7ecf25a8352d5972645c0d444"}
Sep 11 16:16:38.333656 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:38.333635    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9mpsx" event={"ID":"1774723c-dc46-4bdf-b1a9-dc3de662c556","Type":"ContainerStarted","Data":"555b26c82db26b408de0082dfcba4aa9519ec741e88f1fd3a361245912d2e519"}
Sep 11 16:16:38.740293 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:38.740246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret\") pod \"global-pull-secret-syncer-xlpvp\" (UID: \"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f\") " pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:38.740482 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:38.740321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs\") pod \"network-metrics-daemon-z6xtq\" (UID: \"b15c0e16-1077-454c-aafa-16902113243c\") " pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:38.740482 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:38.740465    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:38.740595 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:38.740527    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs podName:b15c0e16-1077-454c-aafa-16902113243c nodeName:}" failed. No retries permitted until 2026-09-11 16:16:40.740508042 +0000 UTC m=+6.080602071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs") pod "network-metrics-daemon-z6xtq" (UID: "b15c0e16-1077-454c-aafa-16902113243c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:38.740656 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:38.740594    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:38.740656 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:38.740627    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret podName:b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f nodeName:}" failed. No retries permitted until 2026-09-11 16:16:40.740616818 +0000 UTC m=+6.080710848 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret") pod "global-pull-secret-syncer-xlpvp" (UID: "b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:38.842089 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:38.841390    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbndv\" (UniqueName: \"kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv\") pod \"network-check-target-qdmr8\" (UID: \"af5da358-605f-4f1f-a20a-5aa63fb8889d\") " pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:38.842089 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:38.841599    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:16:38.842089 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:38.841621    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:16:38.842089 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:38.841635    2561 projected.go:196] Error preparing data for projected volume kube-api-access-mbndv for pod openshift-network-diagnostics/network-check-target-qdmr8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:38.842089 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:38.841696    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv podName:af5da358-605f-4f1f-a20a-5aa63fb8889d nodeName:}" failed. No retries permitted until 2026-09-11 16:16:40.841676309 +0000 UTC m=+6.181770339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbndv" (UniqueName: "kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv") pod "network-check-target-qdmr8" (UID: "af5da358-605f-4f1f-a20a-5aa63fb8889d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:39.272425 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:39.272142    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 16:11:37 +0000 UTC" deadline="2028-03-21 19:05:14.913422857 +0000 UTC"
Sep 11 16:16:39.272425 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:39.272425    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13370h48m35.641011083s"
Sep 11 16:16:39.315627 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:39.315593    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:39.315808 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:39.315728    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:16:39.316226 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:39.316200    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:39.316342 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:39.316308    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:16:39.344237 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:39.343507    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-248.ec2.internal" event={"ID":"e479a5e098468c317c85809b2c11ad9e","Type":"ContainerStarted","Data":"f0ed7ea5d9e0a7631ce0495904a55a6c8ebb3a725c7d0a3c7b3ce011f9ab3e20"}
Sep 11 16:16:39.351350 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:39.350755    2561 generic.go:356] "Generic (PLEG): container finished" podID="bbfd3f90ca8a5cb27eebef2a7ea2cf51" containerID="6ee9f30d7bc42122020db2ae0073b8d839ed3815d250c536e2e0eb3728908eb1" exitCode=0
Sep 11 16:16:39.351350 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:39.350803    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal" event={"ID":"bbfd3f90ca8a5cb27eebef2a7ea2cf51","Type":"ContainerDied","Data":"6ee9f30d7bc42122020db2ae0073b8d839ed3815d250c536e2e0eb3728908eb1"}
Sep 11 16:16:39.383755 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:39.383689    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-248.ec2.internal" podStartSLOduration=3.3836672 podStartE2EDuration="3.3836672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:39.36368619 +0000 UTC m=+4.703780241" watchObservedRunningTime="2026-09-11 16:16:39.3836672 +0000 UTC m=+4.723761250"
Sep 11 16:16:40.315924 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:40.315891    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:40.316557 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:40.316025    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:16:40.376988 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:40.376800    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal" event={"ID":"bbfd3f90ca8a5cb27eebef2a7ea2cf51","Type":"ContainerStarted","Data":"006167665abcc978546dece92c63e6496bc7676a4f856b0922c97f03316b0a79"}
Sep 11 16:16:40.756071 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:40.756016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs\") pod \"network-metrics-daemon-z6xtq\" (UID: \"b15c0e16-1077-454c-aafa-16902113243c\") " pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:40.756264 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:40.756110    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret\") pod \"global-pull-secret-syncer-xlpvp\" (UID: \"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f\") " pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:40.756264 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:40.756244    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:40.756389 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:40.756306    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret podName:b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f nodeName:}" failed. No retries permitted until 2026-09-11 16:16:44.756287356 +0000 UTC m=+10.096381388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret") pod "global-pull-secret-syncer-xlpvp" (UID: "b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:40.756824 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:40.756688    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:40.756824 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:40.756742    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs podName:b15c0e16-1077-454c-aafa-16902113243c nodeName:}" failed. No retries permitted until 2026-09-11 16:16:44.756726779 +0000 UTC m=+10.096820806 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs") pod "network-metrics-daemon-z6xtq" (UID: "b15c0e16-1077-454c-aafa-16902113243c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:40.856608 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:40.856570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbndv\" (UniqueName: \"kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv\") pod \"network-check-target-qdmr8\" (UID: \"af5da358-605f-4f1f-a20a-5aa63fb8889d\") " pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:40.856806 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:40.856787    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:16:40.856875 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:40.856815    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:16:40.856875 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:40.856831    2561 projected.go:196] Error preparing data for projected volume kube-api-access-mbndv for pod openshift-network-diagnostics/network-check-target-qdmr8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:40.856967 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:40.856892    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv podName:af5da358-605f-4f1f-a20a-5aa63fb8889d nodeName:}" failed. No retries permitted until 2026-09-11 16:16:44.856871211 +0000 UTC m=+10.196965249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbndv" (UniqueName: "kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv") pod "network-check-target-qdmr8" (UID: "af5da358-605f-4f1f-a20a-5aa63fb8889d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:41.318888 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:41.318186    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:41.318888 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:41.318376    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:16:41.318888 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:41.318758    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:41.318888 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:41.318852    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:16:42.316339 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:42.316295    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:42.316527 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:42.316446    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:16:43.319381 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:43.319195    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:43.319381 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:43.319216    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:43.319381 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:43.319325    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:16:43.319907 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:43.319412    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:16:44.315852 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:44.315811    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:44.316022 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:44.315952    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:16:44.789419 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:44.789371    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs\") pod \"network-metrics-daemon-z6xtq\" (UID: \"b15c0e16-1077-454c-aafa-16902113243c\") " pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:44.789788 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:44.789474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret\") pod \"global-pull-secret-syncer-xlpvp\" (UID: \"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f\") " pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:44.789788 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:44.789534    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:44.789788 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:44.789558    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:44.789788 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:44.789614    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs podName:b15c0e16-1077-454c-aafa-16902113243c nodeName:}" failed. No retries permitted until 2026-09-11 16:16:52.789596995 +0000 UTC m=+18.129691028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs") pod "network-metrics-daemon-z6xtq" (UID: "b15c0e16-1077-454c-aafa-16902113243c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:44.789788 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:44.789630    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret podName:b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f nodeName:}" failed. No retries permitted until 2026-09-11 16:16:52.789623865 +0000 UTC m=+18.129717890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret") pod "global-pull-secret-syncer-xlpvp" (UID: "b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:44.890847 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:44.890806    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbndv\" (UniqueName: \"kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv\") pod \"network-check-target-qdmr8\" (UID: \"af5da358-605f-4f1f-a20a-5aa63fb8889d\") " pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:44.891038 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:44.890998    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:16:44.891038 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:44.891026    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:16:44.891038 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:44.891040    2561 projected.go:196] Error preparing data for projected volume kube-api-access-mbndv for pod openshift-network-diagnostics/network-check-target-qdmr8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:44.891233 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:44.891106    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv podName:af5da358-605f-4f1f-a20a-5aa63fb8889d nodeName:}" failed. No retries permitted until 2026-09-11 16:16:52.891085832 +0000 UTC m=+18.231179865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbndv" (UniqueName: "kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv") pod "network-check-target-qdmr8" (UID: "af5da358-605f-4f1f-a20a-5aa63fb8889d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:45.316299 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:45.316263    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:45.316463 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:45.316395    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:16:45.316908 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:45.316883    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:45.317040 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:45.317007    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:16:46.316380 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:46.316346    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:46.316770 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:46.316482    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:16:47.316354 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:47.316315    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:47.316554 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:47.316460    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:16:47.316554 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:47.316510    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:47.316922 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:47.316640    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:16:48.315455 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:48.315423    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:48.315592 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:48.315531    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:16:49.316125 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:49.316094    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:49.316596 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:49.316097    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:49.316596 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:49.316239    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:16:49.316596 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:49.316323    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:16:50.316438 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:50.316391    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:50.316828 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:50.316509    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:16:51.316340 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:51.316300    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:51.316515 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:51.316344    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:51.316515 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:51.316437    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:16:51.316928 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:51.316558    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:16:52.315561 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:52.315526    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:52.315779 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:52.315656    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:16:52.849040 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:52.849001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs\") pod \"network-metrics-daemon-z6xtq\" (UID: \"b15c0e16-1077-454c-aafa-16902113243c\") " pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:52.849620 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:52.849079    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret\") pod \"global-pull-secret-syncer-xlpvp\" (UID: \"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f\") " pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:52.849620 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:52.849170    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:52.849620 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:52.849237    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret podName:b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f nodeName:}" failed. No retries permitted until 2026-09-11 16:17:08.849221993 +0000 UTC m=+34.189316021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret") pod "global-pull-secret-syncer-xlpvp" (UID: "b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:16:52.849620 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:52.849184    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:52.849620 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:52.849344    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs podName:b15c0e16-1077-454c-aafa-16902113243c nodeName:}" failed. No retries permitted until 2026-09-11 16:17:08.849322077 +0000 UTC m=+34.189416109 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs") pod "network-metrics-daemon-z6xtq" (UID: "b15c0e16-1077-454c-aafa-16902113243c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:16:52.949646 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:52.949609    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbndv\" (UniqueName: \"kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv\") pod \"network-check-target-qdmr8\" (UID: \"af5da358-605f-4f1f-a20a-5aa63fb8889d\") " pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:52.949818 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:52.949765    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:16:52.949818 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:52.949792    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:16:52.949818 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:52.949805    2561 projected.go:196] Error preparing data for projected volume kube-api-access-mbndv for pod openshift-network-diagnostics/network-check-target-qdmr8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:52.949978 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:52.949868    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv podName:af5da358-605f-4f1f-a20a-5aa63fb8889d nodeName:}" failed. No retries permitted until 2026-09-11 16:17:08.949849618 +0000 UTC m=+34.289943646 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbndv" (UniqueName: "kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv") pod "network-check-target-qdmr8" (UID: "af5da358-605f-4f1f-a20a-5aa63fb8889d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:16:53.316229 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:53.316187    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:53.316433 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:53.316233    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:53.316433 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:53.316342    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:16:53.316549 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:53.316470    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:16:54.316093 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:54.316054    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:54.316609 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:54.316186    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:16:55.316168 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:55.316130    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:55.316636 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:55.316235    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:55.316636 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:55.316255    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:16:55.316636 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:55.316366    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:16:56.315933 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.315905    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:56.316028 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:56.316001    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:16:56.410229 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.410169    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xwsg5" event={"ID":"e86cc7cc-d70a-428d-abc9-4e68716d13d0","Type":"ContainerStarted","Data":"8767a58f19239d60091823b718634f442c0beb30af5e5bd3da1abe788e54f883"}
Sep 11 16:16:56.411673 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.411649    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2lrtq" event={"ID":"f76398de-0983-4cee-afab-0e48e5b8db38","Type":"ContainerStarted","Data":"7a3b72f4580029dab7b5b2dfbce23919bd25b35774886b536929d53ac2d23a2b"}
Sep 11 16:16:56.413149 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.413127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-txkjh" event={"ID":"062d3c49-c0e6-400d-bf42-5ef08d0c83d7","Type":"ContainerStarted","Data":"80a0ba5ee2ff782fdc5dca68e4225d23767b66d117810059050f9d959472725d"}
Sep 11 16:16:56.414537 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.414518    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9mpsx" event={"ID":"1774723c-dc46-4bdf-b1a9-dc3de662c556","Type":"ContainerStarted","Data":"425f2b75e9e45498a4b5f378e059c8c2e23f2e4003e7dec658aa73ff47febe1b"}
Sep 11 16:16:56.416010 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.415989    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xmqw" event={"ID":"53bba805-9236-4e17-bcb2-a4b29b4a2355","Type":"ContainerStarted","Data":"f7bd3854965b7a397813775f6af34d6eefee95974181ee179d83e48fbcd26964"}
Sep 11 16:16:56.417600 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.417578    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w" event={"ID":"d85a376a-563f-4ce7-b0de-6bf847191c83","Type":"ContainerStarted","Data":"63e69fe8791ad51f6b41a18fc024c91005fd1786353a4349adbbdcbd63df52e1"}
Sep 11 16:16:56.419518 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.419501    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:16:56.419867 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.419850    2561 generic.go:356] "Generic (PLEG): container finished" podID="38d0a297-96bb-437e-ac8d-7ef56f3c3c87" containerID="6a8754f47e36a011db3db1f254623a4689b04a901be1db285c79ab1d43f605a8" exitCode=1
Sep 11 16:16:56.419927 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.419876    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg" event={"ID":"38d0a297-96bb-437e-ac8d-7ef56f3c3c87","Type":"ContainerStarted","Data":"0399ed9e4c57f6e415cc00ca69950fbc93c854416cc2fca409746ee37419a5c3"}
Sep 11 16:16:56.419927 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.419924    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg" event={"ID":"38d0a297-96bb-437e-ac8d-7ef56f3c3c87","Type":"ContainerDied","Data":"6a8754f47e36a011db3db1f254623a4689b04a901be1db285c79ab1d43f605a8"}
Sep 11 16:16:56.419995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.419970    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg" event={"ID":"38d0a297-96bb-437e-ac8d-7ef56f3c3c87","Type":"ContainerStarted","Data":"4fe8b95b07f1118b1d116a55981841e29fc770f2dc2e4fa51b623c75d4e89a33"}
Sep 11 16:16:56.445903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.445860    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-248.ec2.internal" podStartSLOduration=20.445826368 podStartE2EDuration="20.445826368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:40.400805379 +0000 UTC m=+5.740899425" watchObservedRunningTime="2026-09-11 16:16:56.445826368 +0000 UTC m=+21.785920406"
Sep 11 16:16:56.498164 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.498102    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xwsg5" podStartSLOduration=3.8501209100000002 podStartE2EDuration="21.498085461s" totalImagesPullingTime="17.647964551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:56.446363976 +0000 UTC m=+21.786458022" watchObservedRunningTime="2026-09-11 16:16:56.498085461 +0000 UTC m=+21.838179507"
Sep 11 16:16:56.498610 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.498577    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-txkjh" podStartSLOduration=3.812844804 podStartE2EDuration="21.498563738s" totalImagesPullingTime="17.685718934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:56.498298075 +0000 UTC m=+21.838392120" watchObservedRunningTime="2026-09-11 16:16:56.498563738 +0000 UTC m=+21.838657778"
Sep 11 16:16:56.594606 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.594554    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2lrtq" podStartSLOduration=3.911347535 podStartE2EDuration="21.594539206s" totalImagesPullingTime="17.683191671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:56.53634958 +0000 UTC m=+21.876443627" watchObservedRunningTime="2026-09-11 16:16:56.594539206 +0000 UTC m=+21.934633252"
Sep 11 16:16:56.638531 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:56.638465    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9mpsx" podStartSLOduration=3.984431714 podStartE2EDuration="21.638444579s" totalImagesPullingTime="17.654012865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:56.598295257 +0000 UTC m=+21.938389303" watchObservedRunningTime="2026-09-11 16:16:56.638444579 +0000 UTC m=+21.978538700"
Sep 11 16:16:57.316123 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:57.316094    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:57.316123 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:57.316133    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:57.316344 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:57.316262    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:16:57.316599 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:57.316538    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:16:57.376810 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:57.376783    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 16:16:57.423792 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:57.423753    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cnprz" event={"ID":"13f82d90-a9ab-4751-affc-6fef32b07c68","Type":"ContainerStarted","Data":"911a250987d754296a85e8c0710c54988c5adbfca0905c6c6542c9753f73a3ed"}
Sep 11 16:16:57.425136 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:57.425106    2561 generic.go:356] "Generic (PLEG): container finished" podID="53bba805-9236-4e17-bcb2-a4b29b4a2355" containerID="f7bd3854965b7a397813775f6af34d6eefee95974181ee179d83e48fbcd26964" exitCode=0
Sep 11 16:16:57.425271 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:57.425204    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xmqw" event={"ID":"53bba805-9236-4e17-bcb2-a4b29b4a2355","Type":"ContainerDied","Data":"f7bd3854965b7a397813775f6af34d6eefee95974181ee179d83e48fbcd26964"}
Sep 11 16:16:57.427163 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:57.427142    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w" event={"ID":"d85a376a-563f-4ce7-b0de-6bf847191c83","Type":"ContainerStarted","Data":"bd0572088a7a62f1296009a7837bdbc28f342e8345d79e59c9c46734689ae0b9"}
Sep 11 16:16:57.430002 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:57.429987    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:16:57.430371 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:57.430353    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg" event={"ID":"38d0a297-96bb-437e-ac8d-7ef56f3c3c87","Type":"ContainerStarted","Data":"2c87ea1e5436442071f48b44e61e254af83a3f0ef4b64d85b109bed6d13322e3"}
Sep 11 16:16:57.430462 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:57.430409    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg" event={"ID":"38d0a297-96bb-437e-ac8d-7ef56f3c3c87","Type":"ContainerStarted","Data":"92be548e2de3bb73b4caf56d60caf4aa108632f758dbe64ecc3c3299c8472771"}
Sep 11 16:16:57.430462 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:57.430441    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg" event={"ID":"38d0a297-96bb-437e-ac8d-7ef56f3c3c87","Type":"ContainerStarted","Data":"485c160809f9a957f0603e7292584853719681a19b10c728828cc640a9390058"}
Sep 11 16:16:57.431760 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:57.431734    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5jvnz" event={"ID":"346d7ed3-37a8-43fd-8544-96f9bc2fb32f","Type":"ContainerStarted","Data":"9df281a3a9c1ed781d3b56b609fcde3f93ef55b9a0c51e1629484cd0f8413ed2"}
Sep 11 16:16:57.440887 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:57.440847    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cnprz" podStartSLOduration=4.7888304680000005 podStartE2EDuration="22.44083475s" totalImagesPullingTime="17.652004282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:57.440808557 +0000 UTC m=+22.780902600" watchObservedRunningTime="2026-09-11 16:16:57.44083475 +0000 UTC m=+22.780928773"
Sep 11 16:16:57.482112 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:57.482043    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5jvnz" podStartSLOduration=4.267477617 podStartE2EDuration="22.482022757s" totalImagesPullingTime="18.21454514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:57.481562513 +0000 UTC m=+22.821656559" watchObservedRunningTime="2026-09-11 16:16:57.482022757 +0000 UTC m=+22.822116801"
Sep 11 16:16:58.290633 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:58.290451    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T16:16:57.376806897Z","UUID":"2f07e1d7-69e7-4206-92d2-37d3dd947be9","Handler":null,"Name":"","Endpoint":""}
Sep 11 16:16:58.293634 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:58.292984    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 16:16:58.293634 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:58.293059    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 16:16:58.316228 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:58.316192    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:16:58.316392 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:58.316298    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:16:58.605931 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:58.605843    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xwsg5"
Sep 11 16:16:58.606573 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:58.606543    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xwsg5"
Sep 11 16:16:59.315459 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:59.315426    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:16:59.315612 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:59.315478    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:16:59.315612 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:59.315569    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:16:59.315743 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:16:59.315709    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:16:59.438641 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:59.438603    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w" event={"ID":"d85a376a-563f-4ce7-b0de-6bf847191c83","Type":"ContainerStarted","Data":"0c5edc5050a906df84a049d712525a4205e8deaffff6efd2c16ffc4a4db556e3"}
Sep 11 16:16:59.442327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:59.442305    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:16:59.442807 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:59.442777    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg" event={"ID":"38d0a297-96bb-437e-ac8d-7ef56f3c3c87","Type":"ContainerStarted","Data":"15cfe34f880247701ada7ac19012ee2f6cd2cff186b72356f813c57d85dfb56e"}
Sep 11 16:16:59.471167 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:16:59.471114    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-glc8w" podStartSLOduration=4.662581707 podStartE2EDuration="24.471091417s" totalImagesPullingTime="19.80850971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:59.470607981 +0000 UTC m=+24.810702026" watchObservedRunningTime="2026-09-11 16:16:59.471091417 +0000 UTC m=+24.811185465"
Sep 11 16:17:00.315795 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:00.315760    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:17:00.316209 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:00.315864    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:17:01.316293 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:01.316256    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:17:01.316293 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:01.316299    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:17:01.317254 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:01.316428    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:17:01.317254 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:01.316557    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:17:01.451509 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:01.451478    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:17:01.451913 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:01.451877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg" event={"ID":"38d0a297-96bb-437e-ac8d-7ef56f3c3c87","Type":"ContainerStarted","Data":"080d9ee8366c03cd79435cf25d6e9bd1d9ae354ca4a2eb3cbc0c2e2a7a1c7e85"}
Sep 11 16:17:01.452294 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:01.452270    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:17:01.452444 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:01.452425    2561 scope.go:117] "RemoveContainer" containerID="6a8754f47e36a011db3db1f254623a4689b04a901be1db285c79ab1d43f605a8"
Sep 11 16:17:01.471008 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:01.470806    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:17:02.315846 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:02.315811    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:17:02.316007 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:02.315932    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:17:02.919488 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:02.919460    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z6xtq"]
Sep 11 16:17:02.919834 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:02.919571    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:17:02.919834 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:02.919668    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:17:02.924019 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:02.923881    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qdmr8"]
Sep 11 16:17:02.924019 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:02.924006    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:17:02.924230 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:02.924143    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:17:02.924522 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:02.924501    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xlpvp"]
Sep 11 16:17:02.924627 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:02.924611    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:17:02.924723 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:02.924705    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:17:03.458874 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:03.458846    2561 generic.go:356] "Generic (PLEG): container finished" podID="53bba805-9236-4e17-bcb2-a4b29b4a2355" containerID="b539f1c4e1c1335de99dfed41c23e14b81e6d19a230b6e0aa21be72ad73dced1" exitCode=0
Sep 11 16:17:03.459056 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:03.458936    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xmqw" event={"ID":"53bba805-9236-4e17-bcb2-a4b29b4a2355","Type":"ContainerDied","Data":"b539f1c4e1c1335de99dfed41c23e14b81e6d19a230b6e0aa21be72ad73dced1"}
Sep 11 16:17:03.462674 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:03.462659    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:17:03.463027 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:03.463007    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg" event={"ID":"38d0a297-96bb-437e-ac8d-7ef56f3c3c87","Type":"ContainerStarted","Data":"ff651d8c66cd0f7c0e6710a54eaefe88634f431519f94d5d1f39ccc3a3cc5d58"}
Sep 11 16:17:03.463291 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:03.463271    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:17:03.463457 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:03.463347    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:17:03.477963 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:03.477934    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:17:04.316317 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:04.316276    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:17:04.316317 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:04.316321    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:17:04.316826 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:04.316319    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:17:04.316826 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:04.316381    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:17:04.316826 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:04.316476    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:17:04.316826 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:04.316556    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:17:04.510289 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:04.510259    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xwsg5"
Sep 11 16:17:04.510455 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:04.510369    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 16:17:04.510869 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:04.510850    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xwsg5"
Sep 11 16:17:04.530098 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:04.530010    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg" podStartSLOduration=11.812478264 podStartE2EDuration="29.529990942s" totalImagesPullingTime="17.717512678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:03.551494137 +0000 UTC m=+28.891588173" watchObservedRunningTime="2026-09-11 16:17:04.529990942 +0000 UTC m=+29.870084986"
Sep 11 16:17:05.469939 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:05.469903    2561 generic.go:356] "Generic (PLEG): container finished" podID="53bba805-9236-4e17-bcb2-a4b29b4a2355" containerID="ea36af1863be4029e643200929339b3bd37948ae1e2ca2fd98f1f8d40d42a561" exitCode=0
Sep 11 16:17:05.470341 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:05.469990    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xmqw" event={"ID":"53bba805-9236-4e17-bcb2-a4b29b4a2355","Type":"ContainerDied","Data":"ea36af1863be4029e643200929339b3bd37948ae1e2ca2fd98f1f8d40d42a561"}
Sep 11 16:17:05.480591 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:05.480554    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg" podUID="38d0a297-96bb-437e-ac8d-7ef56f3c3c87" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 11 16:17:06.316389 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:06.316207    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:17:06.316547 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:06.316246    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:17:06.316547 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:06.316487    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:17:06.316547 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:06.316245    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:17:06.316689 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:06.316539    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:17:06.316689 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:06.316631    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:17:06.475189 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:06.475089    2561 generic.go:356] "Generic (PLEG): container finished" podID="53bba805-9236-4e17-bcb2-a4b29b4a2355" containerID="e8636e2e224f97257fa1b1024129f34232ccc75c8c885d8b54ead15da80ca876" exitCode=0
Sep 11 16:17:06.475189 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:06.475128    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xmqw" event={"ID":"53bba805-9236-4e17-bcb2-a4b29b4a2355","Type":"ContainerDied","Data":"e8636e2e224f97257fa1b1024129f34232ccc75c8c885d8b54ead15da80ca876"}
Sep 11 16:17:08.315550 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:08.315513    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:17:08.315951 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:08.315514    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:17:08.315951 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:08.315622    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xlpvp" podUID="b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f"
Sep 11 16:17:08.315951 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:08.315514    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:17:08.315951 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:08.315696    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qdmr8" podUID="af5da358-605f-4f1f-a20a-5aa63fb8889d"
Sep 11 16:17:08.315951 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:08.315815    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z6xtq" podUID="b15c0e16-1077-454c-aafa-16902113243c"
Sep 11 16:17:08.861230 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:08.861187    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret\") pod \"global-pull-secret-syncer-xlpvp\" (UID: \"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f\") " pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:17:08.861380 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:08.861264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs\") pod \"network-metrics-daemon-z6xtq\" (UID: \"b15c0e16-1077-454c-aafa-16902113243c\") " pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:17:08.861380 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:08.861327    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:08.861471 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:08.861384    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:08.861471 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:08.861429    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret podName:b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f nodeName:}" failed. No retries permitted until 2026-09-11 16:17:40.861407011 +0000 UTC m=+66.201501050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret") pod "global-pull-secret-syncer-xlpvp" (UID: "b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:08.861471 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:08.861454    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs podName:b15c0e16-1077-454c-aafa-16902113243c nodeName:}" failed. No retries permitted until 2026-09-11 16:17:40.861441893 +0000 UTC m=+66.201535923 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs") pod "network-metrics-daemon-z6xtq" (UID: "b15c0e16-1077-454c-aafa-16902113243c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:08.962598 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:08.962552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbndv\" (UniqueName: \"kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv\") pod \"network-check-target-qdmr8\" (UID: \"af5da358-605f-4f1f-a20a-5aa63fb8889d\") " pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:17:08.962754 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:08.962738    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:08.962792 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:08.962759    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:08.962792 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:08.962768    2561 projected.go:196] Error preparing data for projected volume kube-api-access-mbndv for pod openshift-network-diagnostics/network-check-target-qdmr8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:08.962865 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:08.962817    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv podName:af5da358-605f-4f1f-a20a-5aa63fb8889d nodeName:}" failed. No retries permitted until 2026-09-11 16:17:40.962802628 +0000 UTC m=+66.302896652 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mbndv" (UniqueName: "kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv") pod "network-check-target-qdmr8" (UID: "af5da358-605f-4f1f-a20a-5aa63fb8889d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:08.980141 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:08.980114    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-248.ec2.internal" event="NodeReady"
Sep 11 16:17:08.980320 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:08.980307    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 16:17:09.037897 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.037845    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-549b9987dc-qhcjq"]
Sep 11 16:17:09.054087 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.054056    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bnxmw"]
Sep 11 16:17:09.054255 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.054186    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.057156 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.057130    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 16:17:09.057330 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.057293    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 16:17:09.057935 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.057900    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 16:17:09.058250 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.058220    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kb2g5\""
Sep 11 16:17:09.064001 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.063979    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 16:17:09.068153 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.068132    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bnxmw"]
Sep 11 16:17:09.068256 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.068189    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-549b9987dc-qhcjq"]
Sep 11 16:17:09.068312 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.068282    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bnxmw"
Sep 11 16:17:09.070807 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.070789    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qzvwg\""
Sep 11 16:17:09.071108 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.071092    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 16:17:09.071204 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.071115    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 16:17:09.071509 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.071484    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 16:17:09.158081 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.158051    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r7twx"]
Sep 11 16:17:09.163447 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.163422    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da348161-6dda-45e7-b4a4-4a4948eeb2a7-ca-trust-extracted\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.163551 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.163456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnkdj\" (UniqueName: \"kubernetes.io/projected/097bce6a-44cf-4a1d-af40-9681a6ced969-kube-api-access-gnkdj\") pod \"ingress-canary-bnxmw\" (UID: \"097bce6a-44cf-4a1d-af40-9681a6ced969\") " pod="openshift-ingress-canary/ingress-canary-bnxmw"
Sep 11 16:17:09.163551 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.163480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-bound-sa-token\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.163551 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.163532    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/097bce6a-44cf-4a1d-af40-9681a6ced969-cert\") pod \"ingress-canary-bnxmw\" (UID: \"097bce6a-44cf-4a1d-af40-9681a6ced969\") " pod="openshift-ingress-canary/ingress-canary-bnxmw"
Sep 11 16:17:09.163656 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.163566    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-certificates\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.163656 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.163585    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da348161-6dda-45e7-b4a4-4a4948eeb2a7-installation-pull-secrets\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.163656 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.163628    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktttd\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-kube-api-access-ktttd\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.163824 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.163671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.163824 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.163691    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da348161-6dda-45e7-b4a4-4a4948eeb2a7-image-registry-private-configuration\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.163824 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.163741    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da348161-6dda-45e7-b4a4-4a4948eeb2a7-trusted-ca\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.178020 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.177995    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:09.181506 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.181480    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r7twx"]
Sep 11 16:17:09.181613 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.181564    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 16:17:09.181741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.181722    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n6q2x\""
Sep 11 16:17:09.181805 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.181793    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 16:17:09.264476 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.264431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-certificates\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.264660 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.264489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da348161-6dda-45e7-b4a4-4a4948eeb2a7-installation-pull-secrets\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.264660 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.264520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktttd\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-kube-api-access-ktttd\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.264660 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.264549    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:09.264660 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.264617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.264861 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.264677    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5f30a6-4339-4955-baea-2c123c7a1db9-tmp-dir\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:09.264861 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:09.264766    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 16:17:09.264861 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:09.264786    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-549b9987dc-qhcjq: secret "image-registry-tls" not found
Sep 11 16:17:09.264861 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:09.264841    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls podName:da348161-6dda-45e7-b4a4-4a4948eeb2a7 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:09.764822103 +0000 UTC m=+35.104916139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls") pod "image-registry-549b9987dc-qhcjq" (UID: "da348161-6dda-45e7-b4a4-4a4948eeb2a7") : secret "image-registry-tls" not found
Sep 11 16:17:09.265054 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.264780    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da348161-6dda-45e7-b4a4-4a4948eeb2a7-image-registry-private-configuration\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.265054 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.264924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cc5f30a6-4339-4955-baea-2c123c7a1db9-config-volume\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:09.265054 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.264976    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da348161-6dda-45e7-b4a4-4a4948eeb2a7-trusted-ca\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.265054 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.264999    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64w7f\" (UniqueName: \"kubernetes.io/projected/cc5f30a6-4339-4955-baea-2c123c7a1db9-kube-api-access-64w7f\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:09.265054 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.265035    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da348161-6dda-45e7-b4a4-4a4948eeb2a7-ca-trust-extracted\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.265313 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.265063    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnkdj\" (UniqueName: \"kubernetes.io/projected/097bce6a-44cf-4a1d-af40-9681a6ced969-kube-api-access-gnkdj\") pod \"ingress-canary-bnxmw\" (UID: \"097bce6a-44cf-4a1d-af40-9681a6ced969\") " pod="openshift-ingress-canary/ingress-canary-bnxmw"
Sep 11 16:17:09.265313 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.265085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-bound-sa-token\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.265313 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.265117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/097bce6a-44cf-4a1d-af40-9681a6ced969-cert\") pod \"ingress-canary-bnxmw\" (UID: \"097bce6a-44cf-4a1d-af40-9681a6ced969\") " pod="openshift-ingress-canary/ingress-canary-bnxmw"
Sep 11 16:17:09.265313 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.265230    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-certificates\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.265488 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.265396    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da348161-6dda-45e7-b4a4-4a4948eeb2a7-ca-trust-extracted\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.266064 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.265993    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da348161-6dda-45e7-b4a4-4a4948eeb2a7-trusted-ca\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.269126 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.269103    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da348161-6dda-45e7-b4a4-4a4948eeb2a7-installation-pull-secrets\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.269257 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.269149    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da348161-6dda-45e7-b4a4-4a4948eeb2a7-image-registry-private-configuration\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.269257 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.269228    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/097bce6a-44cf-4a1d-af40-9681a6ced969-cert\") pod \"ingress-canary-bnxmw\" (UID: \"097bce6a-44cf-4a1d-af40-9681a6ced969\") " pod="openshift-ingress-canary/ingress-canary-bnxmw"
Sep 11 16:17:09.281396 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.280667    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-bound-sa-token\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.281396 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.281321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktttd\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-kube-api-access-ktttd\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.282252 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.282001    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnkdj\" (UniqueName: \"kubernetes.io/projected/097bce6a-44cf-4a1d-af40-9681a6ced969-kube-api-access-gnkdj\") pod \"ingress-canary-bnxmw\" (UID: \"097bce6a-44cf-4a1d-af40-9681a6ced969\") " pod="openshift-ingress-canary/ingress-canary-bnxmw"
Sep 11 16:17:09.365502 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.365465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:09.366157 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.365514    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5f30a6-4339-4955-baea-2c123c7a1db9-tmp-dir\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:09.366157 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:09.365625    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:09.366157 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.365639    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cc5f30a6-4339-4955-baea-2c123c7a1db9-config-volume\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:09.366157 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:09.365694    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls podName:cc5f30a6-4339-4955-baea-2c123c7a1db9 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:09.865674898 +0000 UTC m=+35.205768922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls") pod "dns-default-r7twx" (UID: "cc5f30a6-4339-4955-baea-2c123c7a1db9") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:09.366157 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.365740    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64w7f\" (UniqueName: \"kubernetes.io/projected/cc5f30a6-4339-4955-baea-2c123c7a1db9-kube-api-access-64w7f\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:09.366157 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.365836    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5f30a6-4339-4955-baea-2c123c7a1db9-tmp-dir\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:09.366445 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.366245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cc5f30a6-4339-4955-baea-2c123c7a1db9-config-volume\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:09.377117 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.377090    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64w7f\" (UniqueName: \"kubernetes.io/projected/cc5f30a6-4339-4955-baea-2c123c7a1db9-kube-api-access-64w7f\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:09.377255 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.377201    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bnxmw"
Sep 11 16:17:09.675371 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.675112    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bnxmw"]
Sep 11 16:17:09.681782 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:17:09.681752    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097bce6a_44cf_4a1d_af40_9681a6ced969.slice/crio-8ccf3a6d2595094394472f0e10efe325482307c7e554137214586b1220dd6480 WatchSource:0}: Error finding container 8ccf3a6d2595094394472f0e10efe325482307c7e554137214586b1220dd6480: Status 404 returned error can't find the container with id 8ccf3a6d2595094394472f0e10efe325482307c7e554137214586b1220dd6480
Sep 11 16:17:09.768089 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.768055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:09.768257 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:09.768235    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 16:17:09.768257 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:09.768254    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-549b9987dc-qhcjq: secret "image-registry-tls" not found
Sep 11 16:17:09.768324 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:09.768317    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls podName:da348161-6dda-45e7-b4a4-4a4948eeb2a7 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:10.768301906 +0000 UTC m=+36.108395929 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls") pod "image-registry-549b9987dc-qhcjq" (UID: "da348161-6dda-45e7-b4a4-4a4948eeb2a7") : secret "image-registry-tls" not found
Sep 11 16:17:09.868476 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:09.868444    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:09.868666 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:09.868610    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:09.868723 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:09.868692    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls podName:cc5f30a6-4339-4955-baea-2c123c7a1db9 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:10.868668858 +0000 UTC m=+36.208762885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls") pod "dns-default-r7twx" (UID: "cc5f30a6-4339-4955-baea-2c123c7a1db9") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:10.316170 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:10.316134    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:17:10.316170 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:10.316170    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:17:10.316425 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:10.316136    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:17:10.320203 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:10.320168    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 16:17:10.320788 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:10.320767    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 16:17:10.320898 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:10.320826    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kx52\""
Sep 11 16:17:10.321040 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:10.320774    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xprtk\""
Sep 11 16:17:10.321138 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:10.321050    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 16:17:10.321138 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:10.321053    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 16:17:10.485858 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:10.485816    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bnxmw" event={"ID":"097bce6a-44cf-4a1d-af40-9681a6ced969","Type":"ContainerStarted","Data":"8ccf3a6d2595094394472f0e10efe325482307c7e554137214586b1220dd6480"}
Sep 11 16:17:10.776241 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:10.776203    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:10.776445 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:10.776347    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 16:17:10.776445 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:10.776367    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-549b9987dc-qhcjq: secret "image-registry-tls" not found
Sep 11 16:17:10.776445 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:10.776443    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls podName:da348161-6dda-45e7-b4a4-4a4948eeb2a7 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:12.776424946 +0000 UTC m=+38.116518969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls") pod "image-registry-549b9987dc-qhcjq" (UID: "da348161-6dda-45e7-b4a4-4a4948eeb2a7") : secret "image-registry-tls" not found
Sep 11 16:17:10.877385 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:10.877347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:10.877571 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:10.877497    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:10.877571 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:10.877569    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls podName:cc5f30a6-4339-4955-baea-2c123c7a1db9 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:12.877548835 +0000 UTC m=+38.217642861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls") pod "dns-default-r7twx" (UID: "cc5f30a6-4339-4955-baea-2c123c7a1db9") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:12.493762 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:12.493727    2561 generic.go:356] "Generic (PLEG): container finished" podID="53bba805-9236-4e17-bcb2-a4b29b4a2355" containerID="8ce16f37cb597496da1e6a605c7bbb86c58ebf62db5893b31cc9efed0bd014bc" exitCode=0
Sep 11 16:17:12.494111 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:12.493776    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xmqw" event={"ID":"53bba805-9236-4e17-bcb2-a4b29b4a2355","Type":"ContainerDied","Data":"8ce16f37cb597496da1e6a605c7bbb86c58ebf62db5893b31cc9efed0bd014bc"}
Sep 11 16:17:12.791484 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:12.791398    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:12.791627 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:12.791551    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 16:17:12.791627 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:12.791571    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-549b9987dc-qhcjq: secret "image-registry-tls" not found
Sep 11 16:17:12.791627 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:12.791626    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls podName:da348161-6dda-45e7-b4a4-4a4948eeb2a7 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:16.791609223 +0000 UTC m=+42.131703246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls") pod "image-registry-549b9987dc-qhcjq" (UID: "da348161-6dda-45e7-b4a4-4a4948eeb2a7") : secret "image-registry-tls" not found
Sep 11 16:17:12.892123 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:12.892092    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:12.892299 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:12.892237    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:12.892299 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:12.892288    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls podName:cc5f30a6-4339-4955-baea-2c123c7a1db9 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:16.892275068 +0000 UTC m=+42.232369092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls") pod "dns-default-r7twx" (UID: "cc5f30a6-4339-4955-baea-2c123c7a1db9") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:13.500276 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:13.500238    2561 generic.go:356] "Generic (PLEG): container finished" podID="53bba805-9236-4e17-bcb2-a4b29b4a2355" containerID="7d63609c4bbe3f0760bd7a073500dc95c2a08c86a95f847efb21b4a9112bd14d" exitCode=0
Sep 11 16:17:13.500671 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:13.500305    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xmqw" event={"ID":"53bba805-9236-4e17-bcb2-a4b29b4a2355","Type":"ContainerDied","Data":"7d63609c4bbe3f0760bd7a073500dc95c2a08c86a95f847efb21b4a9112bd14d"}
Sep 11 16:17:14.504027 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:14.503762    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bnxmw" event={"ID":"097bce6a-44cf-4a1d-af40-9681a6ced969","Type":"ContainerStarted","Data":"16ff599ba6b4753f1fb72f9577038131832e4cefefa3bae8b6708e716bd09733"}
Sep 11 16:17:14.507286 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:14.507258    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xmqw" event={"ID":"53bba805-9236-4e17-bcb2-a4b29b4a2355","Type":"ContainerStarted","Data":"442ba8d91daa6d74db3a02a89b82b065d2605c7de6907b014a1fc95b605f085c"}
Sep 11 16:17:14.524105 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:14.524042    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bnxmw" podStartSLOduration=1.012226821 podStartE2EDuration="5.524019921s" totalImagesPullingTime="4.5117931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:14.523908304 +0000 UTC m=+39.864002359" watchObservedRunningTime="2026-09-11 16:17:14.524019921 +0000 UTC m=+39.864113965"
Sep 11 16:17:14.550594 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:14.550536    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7xmqw" podStartSLOduration=9.41221515 podStartE2EDuration="39.550515227s" totalImagesPullingTime="30.138300077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:14.550379726 +0000 UTC m=+39.890473773" watchObservedRunningTime="2026-09-11 16:17:14.550515227 +0000 UTC m=+39.890609272"
Sep 11 16:17:16.823253 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:16.823197    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:16.823791 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:16.823346    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 16:17:16.823791 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:16.823365    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-549b9987dc-qhcjq: secret "image-registry-tls" not found
Sep 11 16:17:16.823791 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:16.823420    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls podName:da348161-6dda-45e7-b4a4-4a4948eeb2a7 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:24.823405252 +0000 UTC m=+50.163499275 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls") pod "image-registry-549b9987dc-qhcjq" (UID: "da348161-6dda-45e7-b4a4-4a4948eeb2a7") : secret "image-registry-tls" not found
Sep 11 16:17:16.924492 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:16.924452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:16.924626 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:16.924573    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:16.924626 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:16.924624    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls podName:cc5f30a6-4339-4955-baea-2c123c7a1db9 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:24.924610352 +0000 UTC m=+50.264704376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls") pod "dns-default-r7twx" (UID: "cc5f30a6-4339-4955-baea-2c123c7a1db9") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:24.876074 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:24.876025    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:24.876536 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:24.876228    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 16:17:24.876536 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:24.876254    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-549b9987dc-qhcjq: secret "image-registry-tls" not found
Sep 11 16:17:24.876536 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:24.876318    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls podName:da348161-6dda-45e7-b4a4-4a4948eeb2a7 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:40.876301239 +0000 UTC m=+66.216395265 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls") pod "image-registry-549b9987dc-qhcjq" (UID: "da348161-6dda-45e7-b4a4-4a4948eeb2a7") : secret "image-registry-tls" not found
Sep 11 16:17:24.977350 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:24.977307    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:24.977510 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:24.977438    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:24.977510 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:24.977501    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls podName:cc5f30a6-4339-4955-baea-2c123c7a1db9 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:40.977485773 +0000 UTC m=+66.317579798 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls") pod "dns-default-r7twx" (UID: "cc5f30a6-4339-4955-baea-2c123c7a1db9") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:35.480078 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:35.480048    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sz5sg"
Sep 11 16:17:40.893594 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:40.893542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:17:40.893594 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:40.893595    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs\") pod \"network-metrics-daemon-z6xtq\" (UID: \"b15c0e16-1077-454c-aafa-16902113243c\") " pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:17:40.894166 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:40.893634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret\") pod \"global-pull-secret-syncer-xlpvp\" (UID: \"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f\") " pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:17:40.894166 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:40.893736    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 16:17:40.894166 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:40.893764    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-549b9987dc-qhcjq: secret "image-registry-tls" not found
Sep 11 16:17:40.894166 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:40.893845    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls podName:da348161-6dda-45e7-b4a4-4a4948eeb2a7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:12.893822356 +0000 UTC m=+98.233916398 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls") pod "image-registry-549b9987dc-qhcjq" (UID: "da348161-6dda-45e7-b4a4-4a4948eeb2a7") : secret "image-registry-tls" not found
Sep 11 16:17:40.895802 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:40.895778    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 16:17:40.895802 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:40.895794    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 16:17:40.904457 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:40.904438    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 16:17:40.904511 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:40.904500    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs podName:b15c0e16-1077-454c-aafa-16902113243c nodeName:}" failed. No retries permitted until 2026-09-11 16:18:44.904486437 +0000 UTC m=+130.244580460 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs") pod "network-metrics-daemon-z6xtq" (UID: "b15c0e16-1077-454c-aafa-16902113243c") : secret "metrics-daemon-secret" not found
Sep 11 16:17:40.908059 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:40.908038    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f-original-pull-secret\") pod \"global-pull-secret-syncer-xlpvp\" (UID: \"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f\") " pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:17:40.939930 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:40.939900    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xlpvp"
Sep 11 16:17:40.994767 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:40.994736    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:17:40.994908 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:40.994786    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbndv\" (UniqueName: \"kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv\") pod \"network-check-target-qdmr8\" (UID: \"af5da358-605f-4f1f-a20a-5aa63fb8889d\") " pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:17:40.994908 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:40.994884    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:40.998713 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:40.995601    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls podName:cc5f30a6-4339-4955-baea-2c123c7a1db9 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:12.995573332 +0000 UTC m=+98.335667387 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls") pod "dns-default-r7twx" (UID: "cc5f30a6-4339-4955-baea-2c123c7a1db9") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:40.999409 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:40.999386    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 16:17:41.007661 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:41.007514    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 16:17:41.018571 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:41.018544    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbndv\" (UniqueName: \"kubernetes.io/projected/af5da358-605f-4f1f-a20a-5aa63fb8889d-kube-api-access-mbndv\") pod \"network-check-target-qdmr8\" (UID: \"af5da358-605f-4f1f-a20a-5aa63fb8889d\") " pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:17:41.216140 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:41.216111    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xlpvp"]
Sep 11 16:17:41.219792 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:17:41.219765    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27b4250_c0d6_42d8_a2e5_4fc5aa796e2f.slice/crio-031b4434db433b73d4b785ccf7562845bb9210d7204ec28d2f88b76caf88b34e WatchSource:0}: Error finding container 031b4434db433b73d4b785ccf7562845bb9210d7204ec28d2f88b76caf88b34e: Status 404 returned error can't find the container with id 031b4434db433b73d4b785ccf7562845bb9210d7204ec28d2f88b76caf88b34e
Sep 11 16:17:41.236461 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:41.236438    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xprtk\""
Sep 11 16:17:41.244602 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:41.244584    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:17:41.516016 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:41.515944    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qdmr8"]
Sep 11 16:17:41.519404 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:17:41.519378    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5da358_605f_4f1f_a20a_5aa63fb8889d.slice/crio-f716dcd6b7476ac64ab5ad50fa9b91116467325f8e9286ccf427fe05fbd33877 WatchSource:0}: Error finding container f716dcd6b7476ac64ab5ad50fa9b91116467325f8e9286ccf427fe05fbd33877: Status 404 returned error can't find the container with id f716dcd6b7476ac64ab5ad50fa9b91116467325f8e9286ccf427fe05fbd33877
Sep 11 16:17:41.571917 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:41.571882    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qdmr8" event={"ID":"af5da358-605f-4f1f-a20a-5aa63fb8889d","Type":"ContainerStarted","Data":"f716dcd6b7476ac64ab5ad50fa9b91116467325f8e9286ccf427fe05fbd33877"}
Sep 11 16:17:41.572975 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:41.572951    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xlpvp" event={"ID":"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f","Type":"ContainerStarted","Data":"031b4434db433b73d4b785ccf7562845bb9210d7204ec28d2f88b76caf88b34e"}
Sep 11 16:17:47.589464 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:47.589421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qdmr8" event={"ID":"af5da358-605f-4f1f-a20a-5aa63fb8889d","Type":"ContainerStarted","Data":"4f9694b6e7e0cb77b6b1233c58810d688643c325db650aa2c8fe7f570d44aca0"}
Sep 11 16:17:47.589997 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:47.589510    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:17:47.590796 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:47.590774    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xlpvp" event={"ID":"b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f","Type":"ContainerStarted","Data":"4f3ee7f8e3b54732fcc722c74325b01d2b60cc0be46c4ca8fba2669fbee12e82"}
Sep 11 16:17:47.612292 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:47.612236    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qdmr8" podStartSLOduration=67.548851175 podStartE2EDuration="1m12.612215272s" totalImagesPullingTime="5.063364097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:47.611391324 +0000 UTC m=+72.951485370" watchObservedRunningTime="2026-09-11 16:17:47.612215272 +0000 UTC m=+72.952309317"
Sep 11 16:17:47.631755 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:47.631705    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xlpvp" podStartSLOduration=67.257029429 podStartE2EDuration="1m12.631687123s" totalImagesPullingTime="5.374657694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:47.631587976 +0000 UTC m=+72.971682020" watchObservedRunningTime="2026-09-11 16:17:47.631687123 +0000 UTC m=+72.971781166"
Sep 11 16:17:51.364657 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:51.364620    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"]
Sep 11 16:17:51.366807 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:51.366791    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:17:51.372608 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:51.372585    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 11 16:17:51.372608 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:51.372589    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 16:17:51.372820 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:51.372590    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-lzttt\""
Sep 11 16:17:51.372820 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:51.372591    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 16:17:51.372820 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:51.372591    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 11 16:17:51.459705 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:51.459670    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"]
Sep 11 16:17:51.472428 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:51.472388    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2dmxt\" (UID: \"ba15f5ca-5cac-48e7-9622-e3166bdc1dca\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:17:51.472587 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:51.472459    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-2dmxt\" (UID: \"ba15f5ca-5cac-48e7-9622-e3166bdc1dca\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:17:51.472587 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:51.472510    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngg9w\" (UniqueName: \"kubernetes.io/projected/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-kube-api-access-ngg9w\") pod \"cluster-monitoring-operator-6cd954b897-2dmxt\" (UID: \"ba15f5ca-5cac-48e7-9622-e3166bdc1dca\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:17:51.573237 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:51.573199    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2dmxt\" (UID: \"ba15f5ca-5cac-48e7-9622-e3166bdc1dca\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:17:51.573414 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:51.573270    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-2dmxt\" (UID: \"ba15f5ca-5cac-48e7-9622-e3166bdc1dca\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:17:51.573414 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:51.573364    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 16:17:51.573414 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:51.573382    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngg9w\" (UniqueName: \"kubernetes.io/projected/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-kube-api-access-ngg9w\") pod \"cluster-monitoring-operator-6cd954b897-2dmxt\" (UID: \"ba15f5ca-5cac-48e7-9622-e3166bdc1dca\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:17:51.573559 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:51.573462    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls podName:ba15f5ca-5cac-48e7-9622-e3166bdc1dca nodeName:}" failed. No retries permitted until 2026-09-11 16:17:52.073443422 +0000 UTC m=+77.413537446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2dmxt" (UID: "ba15f5ca-5cac-48e7-9622-e3166bdc1dca") : secret "cluster-monitoring-operator-tls" not found
Sep 11 16:17:51.573865 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:51.573846    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-2dmxt\" (UID: \"ba15f5ca-5cac-48e7-9622-e3166bdc1dca\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:17:51.587358 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:51.587330    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngg9w\" (UniqueName: \"kubernetes.io/projected/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-kube-api-access-ngg9w\") pod \"cluster-monitoring-operator-6cd954b897-2dmxt\" (UID: \"ba15f5ca-5cac-48e7-9622-e3166bdc1dca\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:17:52.076535 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:52.076495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2dmxt\" (UID: \"ba15f5ca-5cac-48e7-9622-e3166bdc1dca\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:17:52.076724 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:52.076646    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 16:17:52.076724 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:52.076711    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls podName:ba15f5ca-5cac-48e7-9622-e3166bdc1dca nodeName:}" failed. No retries permitted until 2026-09-11 16:17:53.076692881 +0000 UTC m=+78.416786914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2dmxt" (UID: "ba15f5ca-5cac-48e7-9622-e3166bdc1dca") : secret "cluster-monitoring-operator-tls" not found
Sep 11 16:17:53.083909 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:53.083860    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2dmxt\" (UID: \"ba15f5ca-5cac-48e7-9622-e3166bdc1dca\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:17:53.084388 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:53.084005    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 16:17:53.084388 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:53.084071    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls podName:ba15f5ca-5cac-48e7-9622-e3166bdc1dca nodeName:}" failed. No retries permitted until 2026-09-11 16:17:55.084056821 +0000 UTC m=+80.424150844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2dmxt" (UID: "ba15f5ca-5cac-48e7-9622-e3166bdc1dca") : secret "cluster-monitoring-operator-tls" not found
Sep 11 16:17:55.097441 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.097399    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2dmxt\" (UID: \"ba15f5ca-5cac-48e7-9622-e3166bdc1dca\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:17:55.097810 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:55.097516    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 16:17:55.097810 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:55.097568    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls podName:ba15f5ca-5cac-48e7-9622-e3166bdc1dca nodeName:}" failed. No retries permitted until 2026-09-11 16:17:59.097555615 +0000 UTC m=+84.437649638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2dmxt" (UID: "ba15f5ca-5cac-48e7-9622-e3166bdc1dca") : secret "cluster-monitoring-operator-tls" not found
Sep 11 16:17:55.357676 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.357601    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ttkdv"]
Sep 11 16:17:55.360639 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.360623    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:55.363896 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.363872    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 16:17:55.365212 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.365194    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 16:17:55.365445 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.365429    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 16:17:55.365518 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.365438    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t8wqd\""
Sep 11 16:17:55.365580 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.365436    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 16:17:55.376093 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.376065    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ttkdv"]
Sep 11 16:17:55.399280 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.399250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vbtz\" (UniqueName: \"kubernetes.io/projected/35938125-0fd6-46f4-80f4-04f615985a5b-kube-api-access-8vbtz\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:55.399442 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.399318    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:55.399442 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.399416    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/35938125-0fd6-46f4-80f4-04f615985a5b-data-volume\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:55.399554 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.399448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/35938125-0fd6-46f4-80f4-04f615985a5b-crio-socket\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:55.399554 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.399479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:55.500805 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.500768    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:55.500983 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.500826    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/35938125-0fd6-46f4-80f4-04f615985a5b-data-volume\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:55.500983 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.500854    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/35938125-0fd6-46f4-80f4-04f615985a5b-crio-socket\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:55.500983 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.500875    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:55.500983 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:55.500958    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:17:55.500983 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:55.500963    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:17:55.501141 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.501006    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/35938125-0fd6-46f4-80f4-04f615985a5b-crio-socket\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:55.501141 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:55.501019    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls podName:35938125-0fd6-46f4-80f4-04f615985a5b nodeName:}" failed. No retries permitted until 2026-09-11 16:17:56.001000304 +0000 UTC m=+81.341094336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ttkdv" (UID: "35938125-0fd6-46f4-80f4-04f615985a5b") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:17:55.501141 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:55.501062    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm podName:35938125-0fd6-46f4-80f4-04f615985a5b nodeName:}" failed. No retries permitted until 2026-09-11 16:17:56.001045299 +0000 UTC m=+81.341139323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ttkdv" (UID: "35938125-0fd6-46f4-80f4-04f615985a5b") : configmap "kube-rbac-proxy" not found
Sep 11 16:17:55.501141 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.501078    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vbtz\" (UniqueName: \"kubernetes.io/projected/35938125-0fd6-46f4-80f4-04f615985a5b-kube-api-access-8vbtz\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:55.501298 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.501161    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/35938125-0fd6-46f4-80f4-04f615985a5b-data-volume\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:55.516908 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:55.516874    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vbtz\" (UniqueName: \"kubernetes.io/projected/35938125-0fd6-46f4-80f4-04f615985a5b-kube-api-access-8vbtz\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:56.004758 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:56.004717    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:56.004939 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:56.004795    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:56.004939 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:56.004850    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:17:56.004939 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:56.004896    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:17:56.004939 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:56.004914    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm podName:35938125-0fd6-46f4-80f4-04f615985a5b nodeName:}" failed. No retries permitted until 2026-09-11 16:17:57.004898525 +0000 UTC m=+82.344992552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ttkdv" (UID: "35938125-0fd6-46f4-80f4-04f615985a5b") : configmap "kube-rbac-proxy" not found
Sep 11 16:17:56.005077 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:56.004943    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls podName:35938125-0fd6-46f4-80f4-04f615985a5b nodeName:}" failed. No retries permitted until 2026-09-11 16:17:57.004931608 +0000 UTC m=+82.345025631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ttkdv" (UID: "35938125-0fd6-46f4-80f4-04f615985a5b") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:17:57.011964 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:57.011911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:57.012374 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:57.011983    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:57.012374 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:57.012056    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:17:57.012374 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:57.012073    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:17:57.012374 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:57.012109    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm podName:35938125-0fd6-46f4-80f4-04f615985a5b nodeName:}" failed. No retries permitted until 2026-09-11 16:17:59.012095713 +0000 UTC m=+84.352189736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ttkdv" (UID: "35938125-0fd6-46f4-80f4-04f615985a5b") : configmap "kube-rbac-proxy" not found
Sep 11 16:17:57.012374 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:57.012133    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls podName:35938125-0fd6-46f4-80f4-04f615985a5b nodeName:}" failed. No retries permitted until 2026-09-11 16:17:59.012118927 +0000 UTC m=+84.352212950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ttkdv" (UID: "35938125-0fd6-46f4-80f4-04f615985a5b") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:17:57.620888 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:57.620862    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-txkjh_062d3c49-c0e6-400d-bf42-5ef08d0c83d7/dns-node-resolver/0.log"
Sep 11 16:17:58.234892 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:58.234861    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2lrtq_f76398de-0983-4cee-afab-0e48e5b8db38/node-ca/0.log"
Sep 11 16:17:59.008712 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.008676    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-4txnv"]
Sep 11 16:17:59.012779 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.012763    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-4txnv"
Sep 11 16:17:59.015279 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.015248    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 16:17:59.015400 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.015321    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hhbfc\""
Sep 11 16:17:59.015477 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.015438    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 16:17:59.015477 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.015459    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 16:17:59.015560 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.015480    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 16:17:59.027484 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.027460    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c7251fd5-1785-42cb-a1f4-8041ab427c08-signing-cabundle\") pod \"service-ca-568869fdc5-4txnv\" (UID: \"c7251fd5-1785-42cb-a1f4-8041ab427c08\") " pod="openshift-service-ca/service-ca-568869fdc5-4txnv"
Sep 11 16:17:59.027603 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.027490    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:59.027603 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.027517    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgstx\" (UniqueName: \"kubernetes.io/projected/c7251fd5-1785-42cb-a1f4-8041ab427c08-kube-api-access-xgstx\") pod \"service-ca-568869fdc5-4txnv\" (UID: \"c7251fd5-1785-42cb-a1f4-8041ab427c08\") " pod="openshift-service-ca/service-ca-568869fdc5-4txnv"
Sep 11 16:17:59.027603 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.027580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:17:59.027749 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.027606    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c7251fd5-1785-42cb-a1f4-8041ab427c08-signing-key\") pod \"service-ca-568869fdc5-4txnv\" (UID: \"c7251fd5-1785-42cb-a1f4-8041ab427c08\") " pod="openshift-service-ca/service-ca-568869fdc5-4txnv"
Sep 11 16:17:59.027749 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:59.027622    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:17:59.027749 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:59.027623    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:17:59.027749 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:59.027672    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm podName:35938125-0fd6-46f4-80f4-04f615985a5b nodeName:}" failed. No retries permitted until 2026-09-11 16:18:03.027658394 +0000 UTC m=+88.367752417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ttkdv" (UID: "35938125-0fd6-46f4-80f4-04f615985a5b") : configmap "kube-rbac-proxy" not found
Sep 11 16:17:59.027749 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:59.027699    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls podName:35938125-0fd6-46f4-80f4-04f615985a5b nodeName:}" failed. No retries permitted until 2026-09-11 16:18:03.02767846 +0000 UTC m=+88.367772506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ttkdv" (UID: "35938125-0fd6-46f4-80f4-04f615985a5b") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:17:59.028809 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.028792    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bnxmw_097bce6a-44cf-4a1d-af40-9681a6ced969/serve-healthcheck-canary/0.log"
Sep 11 16:17:59.030285 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.030255    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-4txnv"]
Sep 11 16:17:59.128086 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.128052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c7251fd5-1785-42cb-a1f4-8041ab427c08-signing-key\") pod \"service-ca-568869fdc5-4txnv\" (UID: \"c7251fd5-1785-42cb-a1f4-8041ab427c08\") " pod="openshift-service-ca/service-ca-568869fdc5-4txnv"
Sep 11 16:17:59.128086 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.128090    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2dmxt\" (UID: \"ba15f5ca-5cac-48e7-9622-e3166bdc1dca\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:17:59.128296 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.128131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c7251fd5-1785-42cb-a1f4-8041ab427c08-signing-cabundle\") pod \"service-ca-568869fdc5-4txnv\" (UID: \"c7251fd5-1785-42cb-a1f4-8041ab427c08\") " pod="openshift-service-ca/service-ca-568869fdc5-4txnv"
Sep 11 16:17:59.128296 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.128168    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgstx\" (UniqueName: \"kubernetes.io/projected/c7251fd5-1785-42cb-a1f4-8041ab427c08-kube-api-access-xgstx\") pod \"service-ca-568869fdc5-4txnv\" (UID: \"c7251fd5-1785-42cb-a1f4-8041ab427c08\") " pod="openshift-service-ca/service-ca-568869fdc5-4txnv"
Sep 11 16:17:59.128296 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:59.128255    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 16:17:59.128402 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:17:59.128330    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls podName:ba15f5ca-5cac-48e7-9622-e3166bdc1dca nodeName:}" failed. No retries permitted until 2026-09-11 16:18:07.128314605 +0000 UTC m=+92.468408628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2dmxt" (UID: "ba15f5ca-5cac-48e7-9622-e3166bdc1dca") : secret "cluster-monitoring-operator-tls" not found
Sep 11 16:17:59.128852 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.128825    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c7251fd5-1785-42cb-a1f4-8041ab427c08-signing-cabundle\") pod \"service-ca-568869fdc5-4txnv\" (UID: \"c7251fd5-1785-42cb-a1f4-8041ab427c08\") " pod="openshift-service-ca/service-ca-568869fdc5-4txnv"
Sep 11 16:17:59.130455 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.130437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c7251fd5-1785-42cb-a1f4-8041ab427c08-signing-key\") pod \"service-ca-568869fdc5-4txnv\" (UID: \"c7251fd5-1785-42cb-a1f4-8041ab427c08\") " pod="openshift-service-ca/service-ca-568869fdc5-4txnv"
Sep 11 16:17:59.142383 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.142360    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgstx\" (UniqueName: \"kubernetes.io/projected/c7251fd5-1785-42cb-a1f4-8041ab427c08-kube-api-access-xgstx\") pod \"service-ca-568869fdc5-4txnv\" (UID: \"c7251fd5-1785-42cb-a1f4-8041ab427c08\") " pod="openshift-service-ca/service-ca-568869fdc5-4txnv"
Sep 11 16:17:59.322313 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.322237    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-4txnv"
Sep 11 16:17:59.596820 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.593010    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-4txnv"]
Sep 11 16:17:59.597292 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:17:59.597255    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7251fd5_1785_42cb_a1f4_8041ab427c08.slice/crio-5c52916f0cf1c42caac2457eed62cd329e61a5d2675d10837b3db838c75ef449 WatchSource:0}: Error finding container 5c52916f0cf1c42caac2457eed62cd329e61a5d2675d10837b3db838c75ef449: Status 404 returned error can't find the container with id 5c52916f0cf1c42caac2457eed62cd329e61a5d2675d10837b3db838c75ef449
Sep 11 16:17:59.621425 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:17:59.621391    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-4txnv" event={"ID":"c7251fd5-1785-42cb-a1f4-8041ab427c08","Type":"ContainerStarted","Data":"5c52916f0cf1c42caac2457eed62cd329e61a5d2675d10837b3db838c75ef449"}
Sep 11 16:18:01.627834 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:01.627742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-4txnv" event={"ID":"c7251fd5-1785-42cb-a1f4-8041ab427c08","Type":"ContainerStarted","Data":"1c0f1ba2d511e2439c0c4eaaf7c377c7433b0b62c5bae95466c755d7dd6e28de"}
Sep 11 16:18:01.672121 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:01.672068    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-4txnv" podStartSLOduration=2.064336971 podStartE2EDuration="3.672051704s" totalImagesPullingTime="1.607714733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:01.671540363 +0000 UTC m=+87.011634408" watchObservedRunningTime="2026-09-11 16:18:01.672051704 +0000 UTC m=+87.012145749"
Sep 11 16:18:03.059647 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:03.059603    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:18:03.060123 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:03.059697    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:18:03.060123 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:18:03.059778    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:03.060123 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:18:03.059814    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:03.060123 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:18:03.059857    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls podName:35938125-0fd6-46f4-80f4-04f615985a5b nodeName:}" failed. No retries permitted until 2026-09-11 16:18:11.059836117 +0000 UTC m=+96.399930144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ttkdv" (UID: "35938125-0fd6-46f4-80f4-04f615985a5b") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:03.060123 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:18:03.059877    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm podName:35938125-0fd6-46f4-80f4-04f615985a5b nodeName:}" failed. No retries permitted until 2026-09-11 16:18:11.059867661 +0000 UTC m=+96.399961689 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ttkdv" (UID: "35938125-0fd6-46f4-80f4-04f615985a5b") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:07.192755 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:07.192706    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2dmxt\" (UID: \"ba15f5ca-5cac-48e7-9622-e3166bdc1dca\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:18:07.193122 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:18:07.192863    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 16:18:07.193122 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:18:07.192934    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls podName:ba15f5ca-5cac-48e7-9622-e3166bdc1dca nodeName:}" failed. No retries permitted until 2026-09-11 16:18:23.192918322 +0000 UTC m=+108.533012345 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-2dmxt" (UID: "ba15f5ca-5cac-48e7-9622-e3166bdc1dca") : secret "cluster-monitoring-operator-tls" not found
Sep 11 16:18:11.125794 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:11.125756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:18:11.126168 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:11.125849    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:18:11.126168 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:18:11.126019    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:11.126168 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:18:11.126099    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls podName:35938125-0fd6-46f4-80f4-04f615985a5b nodeName:}" failed. No retries permitted until 2026-09-11 16:18:27.126078442 +0000 UTC m=+112.466172465 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ttkdv" (UID: "35938125-0fd6-46f4-80f4-04f615985a5b") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:11.126349 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:11.126292    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/35938125-0fd6-46f4-80f4-04f615985a5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:18:12.939442 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:12.939405    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:18:12.941850 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:12.941815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls\") pod \"image-registry-549b9987dc-qhcjq\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") " pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:18:12.967932 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:12.967906    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kb2g5\""
Sep 11 16:18:12.976713 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:12.976691    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:18:13.040816 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:13.040776    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:18:13.044092 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:13.044066    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc5f30a6-4339-4955-baea-2c123c7a1db9-metrics-tls\") pod \"dns-default-r7twx\" (UID: \"cc5f30a6-4339-4955-baea-2c123c7a1db9\") " pod="openshift-dns/dns-default-r7twx"
Sep 11 16:18:13.090036 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:13.090006    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n6q2x\""
Sep 11 16:18:13.098164 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:13.098145    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r7twx"
Sep 11 16:18:13.269355 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:13.269270    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-549b9987dc-qhcjq"]
Sep 11 16:18:13.271222 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:13.271197    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda348161_6dda_45e7_b4a4_4a4948eeb2a7.slice/crio-5738207769b75728e90202cdd07a887312dd4555b8d7c82675ce6ec3873ed236 WatchSource:0}: Error finding container 5738207769b75728e90202cdd07a887312dd4555b8d7c82675ce6ec3873ed236: Status 404 returned error can't find the container with id 5738207769b75728e90202cdd07a887312dd4555b8d7c82675ce6ec3873ed236
Sep 11 16:18:13.377212 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:13.377168    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r7twx"]
Sep 11 16:18:13.380930 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:13.380902    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5f30a6_4339_4955_baea_2c123c7a1db9.slice/crio-2964f7c62fecc311762aacd5bb7ba2f3f7699c401dff1353f21099b1effc821e WatchSource:0}: Error finding container 2964f7c62fecc311762aacd5bb7ba2f3f7699c401dff1353f21099b1effc821e: Status 404 returned error can't find the container with id 2964f7c62fecc311762aacd5bb7ba2f3f7699c401dff1353f21099b1effc821e
Sep 11 16:18:13.662467 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:13.662434    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-549b9987dc-qhcjq" event={"ID":"da348161-6dda-45e7-b4a4-4a4948eeb2a7","Type":"ContainerStarted","Data":"5addadf6c7ab20a49a834d836dbd7f3623b303bb76f3694e623352d39551fd83"}
Sep 11 16:18:13.662652 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:13.662489    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-549b9987dc-qhcjq" event={"ID":"da348161-6dda-45e7-b4a4-4a4948eeb2a7","Type":"ContainerStarted","Data":"5738207769b75728e90202cdd07a887312dd4555b8d7c82675ce6ec3873ed236"}
Sep 11 16:18:13.662652 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:13.662510    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:18:13.663618 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:13.663592    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r7twx" event={"ID":"cc5f30a6-4339-4955-baea-2c123c7a1db9","Type":"ContainerStarted","Data":"2964f7c62fecc311762aacd5bb7ba2f3f7699c401dff1353f21099b1effc821e"}
Sep 11 16:18:13.687637 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:13.687525    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-549b9987dc-qhcjq" podStartSLOduration=98.687505375 podStartE2EDuration="1m38.687505375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:13.686622706 +0000 UTC m=+99.026716751" watchObservedRunningTime="2026-09-11 16:18:13.687505375 +0000 UTC m=+99.027599421"
Sep 11 16:18:15.673644 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:15.673606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r7twx" event={"ID":"cc5f30a6-4339-4955-baea-2c123c7a1db9","Type":"ContainerStarted","Data":"e844f967c82647a0afce293acc43cc009b55ec74445ae20b7da81ef3e29eb407"}
Sep 11 16:18:15.674035 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:15.673664    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r7twx" event={"ID":"cc5f30a6-4339-4955-baea-2c123c7a1db9","Type":"ContainerStarted","Data":"d1683bc08ec7df2391627faf6ca353fbb6a0a1a4524c65c7f908d63a4cd6b244"}
Sep 11 16:18:15.674035 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:15.673752    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r7twx"
Sep 11 16:18:18.596793 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:18.596758    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qdmr8"
Sep 11 16:18:18.618759 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:18.618708    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r7twx" podStartSLOduration=68.010129506 podStartE2EDuration="1m9.618691119s" totalImagesPullingTime="1.608561613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:15.695449796 +0000 UTC m=+101.035543840" watchObservedRunningTime="2026-09-11 16:18:18.618691119 +0000 UTC m=+103.958785202"
Sep 11 16:18:20.201609 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.201576    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-kgmlh"]
Sep 11 16:18:20.240564 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.238544    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd"]
Sep 11 16:18:20.241197 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.240890    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kgmlh"
Sep 11 16:18:20.247331 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.247306    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-m44w8\""
Sep 11 16:18:20.247452 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.247329    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 11 16:18:20.247740 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.247611    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 11 16:18:20.262269 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.262246    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-kgmlh"]
Sep 11 16:18:20.262388 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.262277    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd"]
Sep 11 16:18:20.262388 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.262352    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd"
Sep 11 16:18:20.265463 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.265443    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 16:18:20.265950 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.265920    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 16:18:20.265950 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.265937    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 16:18:20.266101 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.265949    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 16:18:20.301680 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.301651    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06ff2972-636e-4ca7-afcb-2c8e1cc76e0d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kgmlh\" (UID: \"06ff2972-636e-4ca7-afcb-2c8e1cc76e0d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kgmlh"
Sep 11 16:18:20.301855 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.301693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/06ff2972-636e-4ca7-afcb-2c8e1cc76e0d-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-kgmlh\" (UID: \"06ff2972-636e-4ca7-afcb-2c8e1cc76e0d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kgmlh"
Sep 11 16:18:20.402446 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.402408    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06ff2972-636e-4ca7-afcb-2c8e1cc76e0d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kgmlh\" (UID: \"06ff2972-636e-4ca7-afcb-2c8e1cc76e0d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kgmlh"
Sep 11 16:18:20.402446 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.402452    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/451d1624-bcae-403c-9973-02f3f1665059-tmp\") pod \"klusterlet-addon-workmgr-5949cb4646-xwvpd\" (UID: \"451d1624-bcae-403c-9973-02f3f1665059\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd"
Sep 11 16:18:20.402689 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.402499    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/06ff2972-636e-4ca7-afcb-2c8e1cc76e0d-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-kgmlh\" (UID: \"06ff2972-636e-4ca7-afcb-2c8e1cc76e0d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kgmlh"
Sep 11 16:18:20.402689 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.402539    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/451d1624-bcae-403c-9973-02f3f1665059-klusterlet-config\") pod \"klusterlet-addon-workmgr-5949cb4646-xwvpd\" (UID: \"451d1624-bcae-403c-9973-02f3f1665059\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd"
Sep 11 16:18:20.402689 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.402558    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jpjm\" (UniqueName: \"kubernetes.io/projected/451d1624-bcae-403c-9973-02f3f1665059-kube-api-access-5jpjm\") pod \"klusterlet-addon-workmgr-5949cb4646-xwvpd\" (UID: \"451d1624-bcae-403c-9973-02f3f1665059\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd"
Sep 11 16:18:20.403194 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.403152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/06ff2972-636e-4ca7-afcb-2c8e1cc76e0d-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-kgmlh\" (UID: \"06ff2972-636e-4ca7-afcb-2c8e1cc76e0d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kgmlh"
Sep 11 16:18:20.404710 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.404681    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06ff2972-636e-4ca7-afcb-2c8e1cc76e0d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-kgmlh\" (UID: \"06ff2972-636e-4ca7-afcb-2c8e1cc76e0d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kgmlh"
Sep 11 16:18:20.503485 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.503401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/451d1624-bcae-403c-9973-02f3f1665059-tmp\") pod \"klusterlet-addon-workmgr-5949cb4646-xwvpd\" (UID: \"451d1624-bcae-403c-9973-02f3f1665059\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd"
Sep 11 16:18:20.503641 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.503566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/451d1624-bcae-403c-9973-02f3f1665059-klusterlet-config\") pod \"klusterlet-addon-workmgr-5949cb4646-xwvpd\" (UID: \"451d1624-bcae-403c-9973-02f3f1665059\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd"
Sep 11 16:18:20.503641 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.503594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jpjm\" (UniqueName: \"kubernetes.io/projected/451d1624-bcae-403c-9973-02f3f1665059-kube-api-access-5jpjm\") pod \"klusterlet-addon-workmgr-5949cb4646-xwvpd\" (UID: \"451d1624-bcae-403c-9973-02f3f1665059\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd"
Sep 11 16:18:20.503814 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.503790    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/451d1624-bcae-403c-9973-02f3f1665059-tmp\") pod \"klusterlet-addon-workmgr-5949cb4646-xwvpd\" (UID: \"451d1624-bcae-403c-9973-02f3f1665059\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd"
Sep 11 16:18:20.506119 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.506092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/451d1624-bcae-403c-9973-02f3f1665059-klusterlet-config\") pod \"klusterlet-addon-workmgr-5949cb4646-xwvpd\" (UID: \"451d1624-bcae-403c-9973-02f3f1665059\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd"
Sep 11 16:18:20.516992 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.516962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jpjm\" (UniqueName: \"kubernetes.io/projected/451d1624-bcae-403c-9973-02f3f1665059-kube-api-access-5jpjm\") pod \"klusterlet-addon-workmgr-5949cb4646-xwvpd\" (UID: \"451d1624-bcae-403c-9973-02f3f1665059\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd"
Sep 11 16:18:20.550546 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.550519    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kgmlh"
Sep 11 16:18:20.571011 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.570983    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd"
Sep 11 16:18:20.890148 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.890068    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-kgmlh"]
Sep 11 16:18:20.893911 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:20.893881    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ff2972_636e_4ca7_afcb_2c8e1cc76e0d.slice/crio-37f729b3be0cfaaca77a12eef80e0180169a8caecde2a2de1ce8f47238133393 WatchSource:0}: Error finding container 37f729b3be0cfaaca77a12eef80e0180169a8caecde2a2de1ce8f47238133393: Status 404 returned error can't find the container with id 37f729b3be0cfaaca77a12eef80e0180169a8caecde2a2de1ce8f47238133393
Sep 11 16:18:20.911574 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:20.911368    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd"]
Sep 11 16:18:20.915088 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:20.915063    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451d1624_bcae_403c_9973_02f3f1665059.slice/crio-0b6a229a1aaa63ecf2e0d2efb712d8145b9fd588ad18b224c04ec9db3878a32b WatchSource:0}: Error finding container 0b6a229a1aaa63ecf2e0d2efb712d8145b9fd588ad18b224c04ec9db3878a32b: Status 404 returned error can't find the container with id 0b6a229a1aaa63ecf2e0d2efb712d8145b9fd588ad18b224c04ec9db3878a32b
Sep 11 16:18:21.695993 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:21.695919    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd" event={"ID":"451d1624-bcae-403c-9973-02f3f1665059","Type":"ContainerStarted","Data":"0b6a229a1aaa63ecf2e0d2efb712d8145b9fd588ad18b224c04ec9db3878a32b"}
Sep 11 16:18:21.698274 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:21.698243    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kgmlh" event={"ID":"06ff2972-636e-4ca7-afcb-2c8e1cc76e0d","Type":"ContainerStarted","Data":"37f729b3be0cfaaca77a12eef80e0180169a8caecde2a2de1ce8f47238133393"}
Sep 11 16:18:22.704761 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:22.704703    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kgmlh" event={"ID":"06ff2972-636e-4ca7-afcb-2c8e1cc76e0d","Type":"ContainerStarted","Data":"53cac0c7f745a954d929cf2cc3698cd4ffd0ab0bc990e7bb3ad0753bcf28dd36"}
Sep 11 16:18:22.743653 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:22.743544    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-kgmlh" podStartSLOduration=1.40957595 podStartE2EDuration="2.743518209s" totalImagesPullingTime="1.333942259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:22.742789089 +0000 UTC m=+108.082883147" watchObservedRunningTime="2026-09-11 16:18:22.743518209 +0000 UTC m=+108.083612254"
Sep 11 16:18:23.227266 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:23.227225    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2dmxt\" (UID: \"ba15f5ca-5cac-48e7-9622-e3166bdc1dca\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:18:23.229980 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:23.229956    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba15f5ca-5cac-48e7-9622-e3166bdc1dca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-2dmxt\" (UID: \"ba15f5ca-5cac-48e7-9622-e3166bdc1dca\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:18:23.479046 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:23.478967    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"
Sep 11 16:18:24.716881 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:24.716787    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd" event={"ID":"451d1624-bcae-403c-9973-02f3f1665059","Type":"ContainerStarted","Data":"69bafc21a03997e2efbe5d42e3c90cc202131c63b8d38ea376a65065eea29d1b"}
Sep 11 16:18:24.717309 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:24.716947    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd"
Sep 11 16:18:24.718581 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:24.718557    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd"
Sep 11 16:18:24.734616 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:24.734592    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt"]
Sep 11 16:18:24.737527 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:24.737497    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba15f5ca_5cac_48e7_9622_e3166bdc1dca.slice/crio-9514593af4eab7b55e22551900b26eafa910ec46bbcdbac688c7a20e7c6bf90d WatchSource:0}: Error finding container 9514593af4eab7b55e22551900b26eafa910ec46bbcdbac688c7a20e7c6bf90d: Status 404 returned error can't find the container with id 9514593af4eab7b55e22551900b26eafa910ec46bbcdbac688c7a20e7c6bf90d
Sep 11 16:18:24.740570 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:24.740529    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5949cb4646-xwvpd" podStartSLOduration=1.117401653 podStartE2EDuration="4.740513968s" totalImagesPullingTime="3.623112315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:24.739360498 +0000 UTC m=+110.079454543" watchObservedRunningTime="2026-09-11 16:18:24.740513968 +0000 UTC m=+110.080608012"
Sep 11 16:18:25.678948 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:25.678917    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r7twx"
Sep 11 16:18:25.721617 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:25.721578    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt" event={"ID":"ba15f5ca-5cac-48e7-9622-e3166bdc1dca","Type":"ContainerStarted","Data":"9514593af4eab7b55e22551900b26eafa910ec46bbcdbac688c7a20e7c6bf90d"}
Sep 11 16:18:26.727399 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:26.727359    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt" event={"ID":"ba15f5ca-5cac-48e7-9622-e3166bdc1dca","Type":"ContainerStarted","Data":"257e5dcaaa8a396e51f9351616952c6cfff1590d77826b73efd1c26be6a14dd9"}
Sep 11 16:18:26.740628 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:26.740596    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h46t5"]
Sep 11 16:18:26.743543 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:26.743524    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h46t5"
Sep 11 16:18:26.745652 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:26.745629    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 16:18:26.745752 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:26.745681    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-p8grh\""
Sep 11 16:18:26.760281 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:26.760257    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h46t5"]
Sep 11 16:18:26.760725 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:26.760677    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-2dmxt" podStartSLOduration=34.33100903 podStartE2EDuration="35.760658755s" totalImagesPullingTime="1.429649725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:26.760375252 +0000 UTC m=+112.100469299" watchObservedRunningTime="2026-09-11 16:18:26.760658755 +0000 UTC m=+112.100752799"
Sep 11 16:18:26.855809 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:26.855771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b13128d0-b6b9-4b72-ae5f-8d638cd5b97c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-h46t5\" (UID: \"b13128d0-b6b9-4b72-ae5f-8d638cd5b97c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h46t5"
Sep 11 16:18:26.956806 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:26.956753    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b13128d0-b6b9-4b72-ae5f-8d638cd5b97c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-h46t5\" (UID: \"b13128d0-b6b9-4b72-ae5f-8d638cd5b97c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h46t5"
Sep 11 16:18:26.959222 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:26.959193    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b13128d0-b6b9-4b72-ae5f-8d638cd5b97c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-h46t5\" (UID: \"b13128d0-b6b9-4b72-ae5f-8d638cd5b97c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h46t5"
Sep 11 16:18:27.052394 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:27.052302    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h46t5"
Sep 11 16:18:27.157760 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:27.157709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:18:27.159964 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:27.159935    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/35938125-0fd6-46f4-80f4-04f615985a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ttkdv\" (UID: \"35938125-0fd6-46f4-80f4-04f615985a5b\") " pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:18:27.169942 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:27.169921    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ttkdv"
Sep 11 16:18:27.322920 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:27.322847    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h46t5"]
Sep 11 16:18:27.325626 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:27.325598    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13128d0_b6b9_4b72_ae5f_8d638cd5b97c.slice/crio-e910c38da2825a0e738738f0a1f7caa09d20e6cd6e99073f301a892eb9428fd9 WatchSource:0}: Error finding container e910c38da2825a0e738738f0a1f7caa09d20e6cd6e99073f301a892eb9428fd9: Status 404 returned error can't find the container with id e910c38da2825a0e738738f0a1f7caa09d20e6cd6e99073f301a892eb9428fd9
Sep 11 16:18:27.443800 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:27.443765    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ttkdv"]
Sep 11 16:18:27.446581 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:27.446550    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35938125_0fd6_46f4_80f4_04f615985a5b.slice/crio-3827b17c196efcb3bcad553335aca0840f95fa016c6610f5e3ada7f2c4c5b56f WatchSource:0}: Error finding container 3827b17c196efcb3bcad553335aca0840f95fa016c6610f5e3ada7f2c4c5b56f: Status 404 returned error can't find the container with id 3827b17c196efcb3bcad553335aca0840f95fa016c6610f5e3ada7f2c4c5b56f
Sep 11 16:18:27.731854 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:27.731820    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h46t5" event={"ID":"b13128d0-b6b9-4b72-ae5f-8d638cd5b97c","Type":"ContainerStarted","Data":"e910c38da2825a0e738738f0a1f7caa09d20e6cd6e99073f301a892eb9428fd9"}
Sep 11 16:18:27.733371 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:27.733347    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ttkdv" event={"ID":"35938125-0fd6-46f4-80f4-04f615985a5b","Type":"ContainerStarted","Data":"b89622255e09a9ea33663b770effa0a03f2aaf17e98ef091b246ef8ccf6410ba"}
Sep 11 16:18:27.733487 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:27.733396    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ttkdv" event={"ID":"35938125-0fd6-46f4-80f4-04f615985a5b","Type":"ContainerStarted","Data":"3827b17c196efcb3bcad553335aca0840f95fa016c6610f5e3ada7f2c4c5b56f"}
Sep 11 16:18:28.738047 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:28.738009    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ttkdv" event={"ID":"35938125-0fd6-46f4-80f4-04f615985a5b","Type":"ContainerStarted","Data":"a73afd5c57731b4e1637f3e369a128494e9636d856a7fc5077e70ad6f921b322"}
Sep 11 16:18:28.739410 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:28.739382    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h46t5" event={"ID":"b13128d0-b6b9-4b72-ae5f-8d638cd5b97c","Type":"ContainerStarted","Data":"12715d4ac22cfca26ad71d1bf1346e5b622941b41ec9b51f52f923fc1f9f4730"}
Sep 11 16:18:28.739588 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:28.739566    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h46t5"
Sep 11 16:18:28.744126 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:28.744103    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h46t5"
Sep 11 16:18:28.760189 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:28.760131    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-h46t5" podStartSLOduration=1.691757256 podStartE2EDuration="2.760117972s" totalImagesPullingTime="1.068360716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:28.759423507 +0000 UTC m=+114.099517552" watchObservedRunningTime="2026-09-11 16:18:28.760117972 +0000 UTC m=+114.100212017"
Sep 11 16:18:29.848972 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:29.848940    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"]
Sep 11 16:18:29.852003 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:29.851986    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"
Sep 11 16:18:29.854583 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:29.854563    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 16:18:29.854583 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:29.854566    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 16:18:29.854814 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:29.854801    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 16:18:29.855249 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:29.855235    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fwzxk\""
Sep 11 16:18:29.864893 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:29.864869    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"]
Sep 11 16:18:29.982342 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:29.982307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3626db7d-bc08-4ba1-b3e8-88eff89e0485-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-tnbqj\" (UID: \"3626db7d-bc08-4ba1-b3e8-88eff89e0485\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"
Sep 11 16:18:29.982483 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:29.982361    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3626db7d-bc08-4ba1-b3e8-88eff89e0485-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-tnbqj\" (UID: \"3626db7d-bc08-4ba1-b3e8-88eff89e0485\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"
Sep 11 16:18:29.982483 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:29.982394    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3626db7d-bc08-4ba1-b3e8-88eff89e0485-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-tnbqj\" (UID: \"3626db7d-bc08-4ba1-b3e8-88eff89e0485\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"
Sep 11 16:18:29.982483 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:29.982419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2zkt\" (UniqueName: \"kubernetes.io/projected/3626db7d-bc08-4ba1-b3e8-88eff89e0485-kube-api-access-r2zkt\") pod \"prometheus-operator-68bcd7f987-tnbqj\" (UID: \"3626db7d-bc08-4ba1-b3e8-88eff89e0485\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"
Sep 11 16:18:30.083022 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:30.082931    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3626db7d-bc08-4ba1-b3e8-88eff89e0485-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-tnbqj\" (UID: \"3626db7d-bc08-4ba1-b3e8-88eff89e0485\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"
Sep 11 16:18:30.083022 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:30.082975    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3626db7d-bc08-4ba1-b3e8-88eff89e0485-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-tnbqj\" (UID: \"3626db7d-bc08-4ba1-b3e8-88eff89e0485\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"
Sep 11 16:18:30.083022 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:30.082992    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2zkt\" (UniqueName: \"kubernetes.io/projected/3626db7d-bc08-4ba1-b3e8-88eff89e0485-kube-api-access-r2zkt\") pod \"prometheus-operator-68bcd7f987-tnbqj\" (UID: \"3626db7d-bc08-4ba1-b3e8-88eff89e0485\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"
Sep 11 16:18:30.083268 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:30.083069    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3626db7d-bc08-4ba1-b3e8-88eff89e0485-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-tnbqj\" (UID: \"3626db7d-bc08-4ba1-b3e8-88eff89e0485\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"
Sep 11 16:18:30.083268 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:18:30.083120    2561 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 11 16:18:30.083268 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:18:30.083239    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3626db7d-bc08-4ba1-b3e8-88eff89e0485-prometheus-operator-tls podName:3626db7d-bc08-4ba1-b3e8-88eff89e0485 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:30.583215271 +0000 UTC m=+115.923309307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/3626db7d-bc08-4ba1-b3e8-88eff89e0485-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-tnbqj" (UID: "3626db7d-bc08-4ba1-b3e8-88eff89e0485") : secret "prometheus-operator-tls" not found
Sep 11 16:18:30.083749 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:30.083731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3626db7d-bc08-4ba1-b3e8-88eff89e0485-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-tnbqj\" (UID: \"3626db7d-bc08-4ba1-b3e8-88eff89e0485\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"
Sep 11 16:18:30.085480 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:30.085451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3626db7d-bc08-4ba1-b3e8-88eff89e0485-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-tnbqj\" (UID: \"3626db7d-bc08-4ba1-b3e8-88eff89e0485\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"
Sep 11 16:18:30.094909 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:30.094884    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2zkt\" (UniqueName: \"kubernetes.io/projected/3626db7d-bc08-4ba1-b3e8-88eff89e0485-kube-api-access-r2zkt\") pod \"prometheus-operator-68bcd7f987-tnbqj\" (UID: \"3626db7d-bc08-4ba1-b3e8-88eff89e0485\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"
Sep 11 16:18:30.586437 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:30.586402    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3626db7d-bc08-4ba1-b3e8-88eff89e0485-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-tnbqj\" (UID: \"3626db7d-bc08-4ba1-b3e8-88eff89e0485\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"
Sep 11 16:18:30.588842 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:30.588823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3626db7d-bc08-4ba1-b3e8-88eff89e0485-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-tnbqj\" (UID: \"3626db7d-bc08-4ba1-b3e8-88eff89e0485\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"
Sep 11 16:18:30.746985 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:30.746940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ttkdv" event={"ID":"35938125-0fd6-46f4-80f4-04f615985a5b","Type":"ContainerStarted","Data":"a65e2c4c93075f9c3e4003a95a5c540e5356b79cff1b94875f27e8d2640a95fc"}
Sep 11 16:18:30.760311 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:30.760289    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"
Sep 11 16:18:30.770574 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:30.770523    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ttkdv" podStartSLOduration=33.566700552 podStartE2EDuration="35.770506287s" totalImagesPullingTime="2.203805735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:30.769090009 +0000 UTC m=+116.109184093" watchObservedRunningTime="2026-09-11 16:18:30.770506287 +0000 UTC m=+116.110600331"
Sep 11 16:18:31.046568 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:31.046529    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj"]
Sep 11 16:18:31.050358 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:31.050323    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3626db7d_bc08_4ba1_b3e8_88eff89e0485.slice/crio-5da7c1798db016dddcc4589e9ab34792cecdcf8df1a233b314b3eab9f58be0d1 WatchSource:0}: Error finding container 5da7c1798db016dddcc4589e9ab34792cecdcf8df1a233b314b3eab9f58be0d1: Status 404 returned error can't find the container with id 5da7c1798db016dddcc4589e9ab34792cecdcf8df1a233b314b3eab9f58be0d1
Sep 11 16:18:31.752227 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:31.752189    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj" event={"ID":"3626db7d-bc08-4ba1-b3e8-88eff89e0485","Type":"ContainerStarted","Data":"5da7c1798db016dddcc4589e9ab34792cecdcf8df1a233b314b3eab9f58be0d1"}
Sep 11 16:18:32.757445 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:32.757411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj" event={"ID":"3626db7d-bc08-4ba1-b3e8-88eff89e0485","Type":"ContainerStarted","Data":"3efc8b8084b219f4dd724797356da5b9e9aee32d3fca6fc72806c090e94dddb3"}
Sep 11 16:18:32.757800 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:32.757463    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj" event={"ID":"3626db7d-bc08-4ba1-b3e8-88eff89e0485","Type":"ContainerStarted","Data":"bfb12142e93c788b282aafae8b49fba1cf2230f85986f8e39a6997c985ab34f6"}
Sep 11 16:18:32.781393 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:32.781338    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-tnbqj" podStartSLOduration=2.710986907 podStartE2EDuration="3.781318186s" totalImagesPullingTime="1.070331279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:32.78045799 +0000 UTC m=+118.120552033" watchObservedRunningTime="2026-09-11 16:18:32.781318186 +0000 UTC m=+118.121412231"
Sep 11 16:18:32.981508 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:32.981475    2561 patch_prober.go:28] interesting pod/image-registry-549b9987dc-qhcjq container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 16:18:32.981668 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:32.981530    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-549b9987dc-qhcjq" podUID="da348161-6dda-45e7-b4a4-4a4948eeb2a7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:18:34.308660 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.308627    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"]
Sep 11 16:18:34.311171 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.311154    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"
Sep 11 16:18:34.314591 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.314569    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 16:18:34.314591 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.314605    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dw29q\""
Sep 11 16:18:34.314804 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.314724    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 16:18:34.332727 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.332699    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"]
Sep 11 16:18:34.419518 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.419479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-xhkkp\" (UID: \"ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"
Sep 11 16:18:34.419717 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.419591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-xhkkp\" (UID: \"ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"
Sep 11 16:18:34.419717 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.419616    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv2l4\" (UniqueName: \"kubernetes.io/projected/ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2-kube-api-access-cv2l4\") pod \"openshift-state-metrics-67dcf8c979-xhkkp\" (UID: \"ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"
Sep 11 16:18:34.419717 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.419645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-xhkkp\" (UID: \"ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"
Sep 11 16:18:34.437960 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.437918    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"]
Sep 11 16:18:34.440032 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.440013    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.444526 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.444498    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 11 16:18:34.445525 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.445509    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 11 16:18:34.445791 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.445774    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tw2bj\""
Sep 11 16:18:34.453985 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.453961    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 11 16:18:34.511135 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.511101    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"]
Sep 11 16:18:34.521018 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.520974    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-xhkkp\" (UID: \"ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"
Sep 11 16:18:34.521163 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.521023    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv2l4\" (UniqueName: \"kubernetes.io/projected/ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2-kube-api-access-cv2l4\") pod \"openshift-state-metrics-67dcf8c979-xhkkp\" (UID: \"ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"
Sep 11 16:18:34.521163 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.521056    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87a63b67-349c-46c4-8123-c2413ceafd6f-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.521163 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.521086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qpqj\" (UniqueName: \"kubernetes.io/projected/87a63b67-349c-46c4-8123-c2413ceafd6f-kube-api-access-5qpqj\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.521163 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.521118    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-xhkkp\" (UID: \"ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"
Sep 11 16:18:34.521395 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.521302    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87a63b67-349c-46c4-8123-c2413ceafd6f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.521395 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.521357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87a63b67-349c-46c4-8123-c2413ceafd6f-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.521395 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.521390    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-xhkkp\" (UID: \"ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"
Sep 11 16:18:34.521535 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.521500    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87a63b67-349c-46c4-8123-c2413ceafd6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.521587 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.521555    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87a63b67-349c-46c4-8123-c2413ceafd6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.522271 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.522249    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-xhkkp\" (UID: \"ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"
Sep 11 16:18:34.523750 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.523725    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-xhkkp\" (UID: \"ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"
Sep 11 16:18:34.524290 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.524272    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-xhkkp\" (UID: \"ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"
Sep 11 16:18:34.538657 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.538625    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv2l4\" (UniqueName: \"kubernetes.io/projected/ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2-kube-api-access-cv2l4\") pod \"openshift-state-metrics-67dcf8c979-xhkkp\" (UID: \"ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"
Sep 11 16:18:34.543004 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.542971    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xlhbf"]
Sep 11 16:18:34.545339 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.545320    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.548483 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.548460    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7f2r7\""
Sep 11 16:18:34.562762 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.562708    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 16:18:34.569567 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.569545    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 16:18:34.569656 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.569600    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 16:18:34.622749 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.622719    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"
Sep 11 16:18:34.622928 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.622907    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpkzm\" (UniqueName: \"kubernetes.io/projected/6084648b-e681-43b3-b0ea-28a6efef0b7c-kube-api-access-zpkzm\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.622997 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.622943    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87a63b67-349c-46c4-8123-c2413ceafd6f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.622997 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.622966    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6084648b-e681-43b3-b0ea-28a6efef0b7c-root\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.623066 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.623009    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87a63b67-349c-46c4-8123-c2413ceafd6f-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.623066 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.623032    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6084648b-e681-43b3-b0ea-28a6efef0b7c-node-exporter-textfile\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.623149 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.623068    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6084648b-e681-43b3-b0ea-28a6efef0b7c-metrics-client-ca\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.623149 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.623099    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87a63b67-349c-46c4-8123-c2413ceafd6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.623149 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.623124    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6084648b-e681-43b3-b0ea-28a6efef0b7c-node-exporter-accelerators-collector-config\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.623296 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.623157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6084648b-e681-43b3-b0ea-28a6efef0b7c-node-exporter-wtmp\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.623296 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.623241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87a63b67-349c-46c4-8123-c2413ceafd6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.623296 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.623282    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6084648b-e681-43b3-b0ea-28a6efef0b7c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.623413 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.623331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6084648b-e681-43b3-b0ea-28a6efef0b7c-sys\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.623413 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.623358    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6084648b-e681-43b3-b0ea-28a6efef0b7c-node-exporter-tls\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.623510 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.623467    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87a63b67-349c-46c4-8123-c2413ceafd6f-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.623557 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.623505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qpqj\" (UniqueName: \"kubernetes.io/projected/87a63b67-349c-46c4-8123-c2413ceafd6f-kube-api-access-5qpqj\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.623892 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.623863    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87a63b67-349c-46c4-8123-c2413ceafd6f-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.624015 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.623866    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/87a63b67-349c-46c4-8123-c2413ceafd6f-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.624015 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.623868    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/87a63b67-349c-46c4-8123-c2413ceafd6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.626106 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.626079    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/87a63b67-349c-46c4-8123-c2413ceafd6f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.626228 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.626198    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/87a63b67-349c-46c4-8123-c2413ceafd6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.640832 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.640803    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qpqj\" (UniqueName: \"kubernetes.io/projected/87a63b67-349c-46c4-8123-c2413ceafd6f-kube-api-access-5qpqj\") pod \"kube-state-metrics-59fbdd77d-88cgp\" (UID: \"87a63b67-349c-46c4-8123-c2413ceafd6f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.677545 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.677515    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:18:34.725202 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.724650    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6084648b-e681-43b3-b0ea-28a6efef0b7c-metrics-client-ca\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.725202 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.724696    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6084648b-e681-43b3-b0ea-28a6efef0b7c-node-exporter-accelerators-collector-config\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.725202 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.724724    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6084648b-e681-43b3-b0ea-28a6efef0b7c-node-exporter-wtmp\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.725202 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.724760    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6084648b-e681-43b3-b0ea-28a6efef0b7c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.725202 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.724804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6084648b-e681-43b3-b0ea-28a6efef0b7c-sys\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.725202 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.724825    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6084648b-e681-43b3-b0ea-28a6efef0b7c-node-exporter-tls\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.725202 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.724867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpkzm\" (UniqueName: \"kubernetes.io/projected/6084648b-e681-43b3-b0ea-28a6efef0b7c-kube-api-access-zpkzm\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.725202 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.724899    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6084648b-e681-43b3-b0ea-28a6efef0b7c-root\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.725202 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.724926    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6084648b-e681-43b3-b0ea-28a6efef0b7c-node-exporter-textfile\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.725202 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.725204    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6084648b-e681-43b3-b0ea-28a6efef0b7c-node-exporter-accelerators-collector-config\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.725786 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.725267    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6084648b-e681-43b3-b0ea-28a6efef0b7c-sys\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.725786 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.725313    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6084648b-e681-43b3-b0ea-28a6efef0b7c-metrics-client-ca\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.725786 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.725349    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6084648b-e681-43b3-b0ea-28a6efef0b7c-node-exporter-wtmp\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.725786 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.725714    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6084648b-e681-43b3-b0ea-28a6efef0b7c-root\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.725985 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.725826    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6084648b-e681-43b3-b0ea-28a6efef0b7c-node-exporter-textfile\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.728146 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.728120    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6084648b-e681-43b3-b0ea-28a6efef0b7c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.728275 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.728122    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6084648b-e681-43b3-b0ea-28a6efef0b7c-node-exporter-tls\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.736403 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.736354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpkzm\" (UniqueName: \"kubernetes.io/projected/6084648b-e681-43b3-b0ea-28a6efef0b7c-kube-api-access-zpkzm\") pod \"node-exporter-xlhbf\" (UID: \"6084648b-e681-43b3-b0ea-28a6efef0b7c\") " pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.751431 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.751393    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"
Sep 11 16:18:34.854211 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.854112    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xlhbf"
Sep 11 16:18:34.864234 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:34.864197    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6084648b_e681_43b3_b0ea_28a6efef0b7c.slice/crio-d808a063645210dc9fa84138f89317a36120c648f60a01cd2740a169bbfb9873 WatchSource:0}: Error finding container d808a063645210dc9fa84138f89317a36120c648f60a01cd2740a169bbfb9873: Status 404 returned error can't find the container with id d808a063645210dc9fa84138f89317a36120c648f60a01cd2740a169bbfb9873
Sep 11 16:18:34.916611 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:34.916470    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp"]
Sep 11 16:18:34.919398 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:34.919373    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba013ca8_ec3b_4ce8_940d_620cb9c7d3a2.slice/crio-4ea7cf26a0c8d146bd617e689a60f51050725f314f7bdfd3563e22035d74ad81 WatchSource:0}: Error finding container 4ea7cf26a0c8d146bd617e689a60f51050725f314f7bdfd3563e22035d74ad81: Status 404 returned error can't find the container with id 4ea7cf26a0c8d146bd617e689a60f51050725f314f7bdfd3563e22035d74ad81
Sep 11 16:18:35.040485 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.040456    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp"]
Sep 11 16:18:35.044000 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:35.043934    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a63b67_349c_46c4_8123_c2413ceafd6f.slice/crio-b4997756ee78758c4b497867c0062709d467a63ed4576f63a1ff001aafc8ea00 WatchSource:0}: Error finding container b4997756ee78758c4b497867c0062709d467a63ed4576f63a1ff001aafc8ea00: Status 404 returned error can't find the container with id b4997756ee78758c4b497867c0062709d467a63ed4576f63a1ff001aafc8ea00
Sep 11 16:18:35.454098 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.454052    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:18:35.461974 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.461948    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.464449 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.464399    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 16:18:35.465066 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.465044    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zb8g5\""
Sep 11 16:18:35.465205 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.465192    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 16:18:35.465303 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.465274    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 16:18:35.465711 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.465684    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 16:18:35.465977 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.465958    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 16:18:35.466212 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.466196    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 16:18:35.466742 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.466603    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 16:18:35.466742 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.466629    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 16:18:35.466742 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.466657    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 16:18:35.479184 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.479133    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:18:35.634162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.633601    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0340401-505e-41ab-b9c0-fff1f0bc4493-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.634162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.633650    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0340401-505e-41ab-b9c0-fff1f0bc4493-config-out\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.634162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.633677    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25czd\" (UniqueName: \"kubernetes.io/projected/c0340401-505e-41ab-b9c0-fff1f0bc4493-kube-api-access-25czd\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.634162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.633716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.634162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.633742    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0340401-505e-41ab-b9c0-fff1f0bc4493-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.634162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.633767    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0340401-505e-41ab-b9c0-fff1f0bc4493-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.634162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.633794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.634162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.633816    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.634162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.633862    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.634162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.633887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-web-config\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.634162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.633942    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.634162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.633969    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0340401-505e-41ab-b9c0-fff1f0bc4493-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.634162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.633993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.735368 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.735262    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.735368 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.735308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-web-config\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.735584 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.735373    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.735584 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.735407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0340401-505e-41ab-b9c0-fff1f0bc4493-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.735584 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.735431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.735584 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.735463    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0340401-505e-41ab-b9c0-fff1f0bc4493-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.735584 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.735491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0340401-505e-41ab-b9c0-fff1f0bc4493-config-out\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.735584 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.735514    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25czd\" (UniqueName: \"kubernetes.io/projected/c0340401-505e-41ab-b9c0-fff1f0bc4493-kube-api-access-25czd\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.735584 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.735557    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.735584 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.735585    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0340401-505e-41ab-b9c0-fff1f0bc4493-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.735963 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.735613    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0340401-505e-41ab-b9c0-fff1f0bc4493-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.735963 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.735643    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.735963 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.735664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.739997 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.737532    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0340401-505e-41ab-b9c0-fff1f0bc4493-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.739997 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.738247    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0340401-505e-41ab-b9c0-fff1f0bc4493-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.739997 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.738850    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0340401-505e-41ab-b9c0-fff1f0bc4493-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.742479 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.742450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0340401-505e-41ab-b9c0-fff1f0bc4493-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.747391 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.745535    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25czd\" (UniqueName: \"kubernetes.io/projected/c0340401-505e-41ab-b9c0-fff1f0bc4493-kube-api-access-25czd\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.747391 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.745617    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0340401-505e-41ab-b9c0-fff1f0bc4493-config-out\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.747391 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.745737    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.747391 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.745965    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-web-config\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.747391 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.746407    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.747391 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.746922    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.747391 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.747307    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.747391 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.747356    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.748310 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.748274    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.771991 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.771953    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xlhbf" event={"ID":"6084648b-e681-43b3-b0ea-28a6efef0b7c","Type":"ContainerStarted","Data":"d808a063645210dc9fa84138f89317a36120c648f60a01cd2740a169bbfb9873"}
Sep 11 16:18:35.775267 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.775230    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp" event={"ID":"ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2","Type":"ContainerStarted","Data":"a59bdf10f112c01294387d432482b2569beacba4a4b4a82fb249270519d35fd6"}
Sep 11 16:18:35.775422 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.775299    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp" event={"ID":"ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2","Type":"ContainerStarted","Data":"17e271202d85e1f73031bf06e0ba77512e9ed60c59a0d8089abf0debecce0ded"}
Sep 11 16:18:35.775422 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.775328    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp" event={"ID":"ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2","Type":"ContainerStarted","Data":"4ea7cf26a0c8d146bd617e689a60f51050725f314f7bdfd3563e22035d74ad81"}
Sep 11 16:18:35.776805 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.776782    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:18:35.777205 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:35.777116    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp" event={"ID":"87a63b67-349c-46c4-8123-c2413ceafd6f","Type":"ContainerStarted","Data":"b4997756ee78758c4b497867c0062709d467a63ed4576f63a1ff001aafc8ea00"}
Sep 11 16:18:36.783348 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:36.783310    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp" event={"ID":"87a63b67-349c-46c4-8123-c2413ceafd6f","Type":"ContainerStarted","Data":"1b5d4d525de08be76676b4acfe0451709fafd710b8472039a652919e0a44b49e"}
Sep 11 16:18:36.783779 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:36.783404    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp" event={"ID":"87a63b67-349c-46c4-8123-c2413ceafd6f","Type":"ContainerStarted","Data":"b5684f6112a11563e06f7de654021d95eb3e24b7c3ae2a04d46f37c4d643398d"}
Sep 11 16:18:36.783779 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:36.783437    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp" event={"ID":"87a63b67-349c-46c4-8123-c2413ceafd6f","Type":"ContainerStarted","Data":"6e959e458d41591b4e54bd384d96040026f7a796af5f0be659a621b6fcf814fa"}
Sep 11 16:18:36.784986 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:36.784949    2561 generic.go:356] "Generic (PLEG): container finished" podID="6084648b-e681-43b3-b0ea-28a6efef0b7c" containerID="c9d8ab99b4a2312479f27db16875454f7ad305696424424c8a000b5a8bff54d4" exitCode=0
Sep 11 16:18:36.785091 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:36.784994    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xlhbf" event={"ID":"6084648b-e681-43b3-b0ea-28a6efef0b7c","Type":"ContainerDied","Data":"c9d8ab99b4a2312479f27db16875454f7ad305696424424c8a000b5a8bff54d4"}
Sep 11 16:18:36.787611 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:36.787548    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp" event={"ID":"ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2","Type":"ContainerStarted","Data":"931c854234676c2eda6de5934c4bc81c2243b3abaa15394350c9408b68b604d3"}
Sep 11 16:18:36.807526 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:36.806856    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:18:36.808630 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:36.808596    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0340401_505e_41ab_b9c0_fff1f0bc4493.slice/crio-7ee7a9493fbdb86c5e96b617ac8f423b193a2faaf686c8db00fbc527cf2f5709 WatchSource:0}: Error finding container 7ee7a9493fbdb86c5e96b617ac8f423b193a2faaf686c8db00fbc527cf2f5709: Status 404 returned error can't find the container with id 7ee7a9493fbdb86c5e96b617ac8f423b193a2faaf686c8db00fbc527cf2f5709
Sep 11 16:18:36.823584 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:36.823536    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-88cgp" podStartSLOduration=1.357966626 podStartE2EDuration="2.823519074s" totalImagesPullingTime="1.465552448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:36.822414304 +0000 UTC m=+122.162508348" watchObservedRunningTime="2026-09-11 16:18:36.823519074 +0000 UTC m=+122.163613118"
Sep 11 16:18:36.850992 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:36.850941    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-xhkkp" podStartSLOduration=1.3756874190000001 podStartE2EDuration="2.850923094s" totalImagesPullingTime="1.475235675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:36.849216171 +0000 UTC m=+122.189310217" watchObservedRunningTime="2026-09-11 16:18:36.850923094 +0000 UTC m=+122.191017139"
Sep 11 16:18:37.502225 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.502185    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"]
Sep 11 16:18:37.505785 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.505760    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.509475 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.509446    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 16:18:37.509614 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.509574    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8p31jsh7s0jr1\""
Sep 11 16:18:37.509952 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.509833    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 16:18:37.509952 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.509846    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vlmzh\""
Sep 11 16:18:37.509952 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.509846    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 16:18:37.509952 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.509887    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 16:18:37.509952 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.509901    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 16:18:37.525748 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.525723    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"]
Sep 11 16:18:37.655834 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.655738    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.655834 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.655801    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.656020 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.655877    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b15f23b5-8a1e-4763-838c-ba6550c4918d-metrics-client-ca\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.656020 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.655923    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.656020 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.655953    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-thanos-querier-tls\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.656020 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.655981    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-grpc-tls\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.656259 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.656053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.656259 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.656126    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjrd\" (UniqueName: \"kubernetes.io/projected/b15f23b5-8a1e-4763-838c-ba6550c4918d-kube-api-access-lkjrd\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.757566 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.757486    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b15f23b5-8a1e-4763-838c-ba6550c4918d-metrics-client-ca\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.757566 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.757545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.757791 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.757577    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-thanos-querier-tls\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.757791 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.757606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-grpc-tls\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.757791 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.757705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.757791 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.757755    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjrd\" (UniqueName: \"kubernetes.io/projected/b15f23b5-8a1e-4763-838c-ba6550c4918d-kube-api-access-lkjrd\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.757791 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.757788    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.758026 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.757823    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.758525 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.758465    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b15f23b5-8a1e-4763-838c-ba6550c4918d-metrics-client-ca\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.760961 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.760933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.761295 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.761273    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.761472 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.761452    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-grpc-tls\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.761525 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.761452    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-thanos-querier-tls\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.761525 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.761497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.761772 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.761706    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b15f23b5-8a1e-4763-838c-ba6550c4918d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.767281 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.767229    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjrd\" (UniqueName: \"kubernetes.io/projected/b15f23b5-8a1e-4763-838c-ba6550c4918d-kube-api-access-lkjrd\") pod \"thanos-querier-6c75b4f8f-xqr2k\" (UID: \"b15f23b5-8a1e-4763-838c-ba6550c4918d\") " pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.793760 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.793709    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0340401-505e-41ab-b9c0-fff1f0bc4493","Type":"ContainerStarted","Data":"7ee7a9493fbdb86c5e96b617ac8f423b193a2faaf686c8db00fbc527cf2f5709"}
Sep 11 16:18:37.796564 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.796529    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xlhbf" event={"ID":"6084648b-e681-43b3-b0ea-28a6efef0b7c","Type":"ContainerStarted","Data":"21a7534126052fffb76b731fa6d53ad06104cce7cb9bc9141747c62bab980b53"}
Sep 11 16:18:37.796771 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.796594    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xlhbf" event={"ID":"6084648b-e681-43b3-b0ea-28a6efef0b7c","Type":"ContainerStarted","Data":"97d924f9662925e510b160b3678e713805c43a8d3db405b5dea241c8b46ec467"}
Sep 11 16:18:37.815368 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.815300    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:37.832220 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:37.832068    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xlhbf" podStartSLOduration=2.102720889 podStartE2EDuration="3.832045176s" totalImagesPullingTime="1.643399914s" totalInitContainerRuntime="85.924373ms" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:37.829307262 +0000 UTC m=+123.169401309" watchObservedRunningTime="2026-09-11 16:18:37.832045176 +0000 UTC m=+123.172139222"
Sep 11 16:18:38.115612 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:38.115536    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"]
Sep 11 16:18:38.135507 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:38.135481    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15f23b5_8a1e_4763_838c_ba6550c4918d.slice/crio-74e97c08ff4e8494f54a7dd523da3db6cefa0af176bf9f066071dd6977e52dd3 WatchSource:0}: Error finding container 74e97c08ff4e8494f54a7dd523da3db6cefa0af176bf9f066071dd6977e52dd3: Status 404 returned error can't find the container with id 74e97c08ff4e8494f54a7dd523da3db6cefa0af176bf9f066071dd6977e52dd3
Sep 11 16:18:38.805591 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:38.805554    2561 generic.go:356] "Generic (PLEG): container finished" podID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerID="6817fd748c356811980e7a0bd4bb70c8428f26102d7c315f1c6dc47f09a0b82e" exitCode=0
Sep 11 16:18:38.806016 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:38.805646    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0340401-505e-41ab-b9c0-fff1f0bc4493","Type":"ContainerDied","Data":"6817fd748c356811980e7a0bd4bb70c8428f26102d7c315f1c6dc47f09a0b82e"}
Sep 11 16:18:38.812143 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:38.809111    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k" event={"ID":"b15f23b5-8a1e-4763-838c-ba6550c4918d","Type":"ContainerStarted","Data":"74e97c08ff4e8494f54a7dd523da3db6cefa0af176bf9f066071dd6977e52dd3"}
Sep 11 16:18:39.049922 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.049888    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9"]
Sep 11 16:18:39.053620 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.053590    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9"
Sep 11 16:18:39.056342 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.055787    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 16:18:39.056342 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.056113    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lqkxw\""
Sep 11 16:18:39.071767 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.071737    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9"]
Sep 11 16:18:39.174268 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.174232    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b08d9859-821d-4f5f-8058-eaf19c990774-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-49rm9\" (UID: \"b08d9859-821d-4f5f-8058-eaf19c990774\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9"
Sep 11 16:18:39.174447 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.174424    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h84m9\" (UniqueName: \"kubernetes.io/projected/b08d9859-821d-4f5f-8058-eaf19c990774-kube-api-access-h84m9\") pod \"monitoring-plugin-5d85778b9d-49rm9\" (UID: \"b08d9859-821d-4f5f-8058-eaf19c990774\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9"
Sep 11 16:18:39.275754 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.275670    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b08d9859-821d-4f5f-8058-eaf19c990774-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-49rm9\" (UID: \"b08d9859-821d-4f5f-8058-eaf19c990774\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9"
Sep 11 16:18:39.275940 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.275805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h84m9\" (UniqueName: \"kubernetes.io/projected/b08d9859-821d-4f5f-8058-eaf19c990774-kube-api-access-h84m9\") pod \"monitoring-plugin-5d85778b9d-49rm9\" (UID: \"b08d9859-821d-4f5f-8058-eaf19c990774\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9"
Sep 11 16:18:39.275940 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:18:39.275822    2561 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 11 16:18:39.275940 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:18:39.275882    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b08d9859-821d-4f5f-8058-eaf19c990774-monitoring-plugin-cert podName:b08d9859-821d-4f5f-8058-eaf19c990774 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:39.775864504 +0000 UTC m=+125.115958538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/b08d9859-821d-4f5f-8058-eaf19c990774-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-49rm9" (UID: "b08d9859-821d-4f5f-8058-eaf19c990774") : secret "monitoring-plugin-cert" not found
Sep 11 16:18:39.289225 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.289187    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h84m9\" (UniqueName: \"kubernetes.io/projected/b08d9859-821d-4f5f-8058-eaf19c990774-kube-api-access-h84m9\") pod \"monitoring-plugin-5d85778b9d-49rm9\" (UID: \"b08d9859-821d-4f5f-8058-eaf19c990774\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9"
Sep 11 16:18:39.551934 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.551869    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-68d6c68898-tstt8"]
Sep 11 16:18:39.555843 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.555757    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.558142 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.558116    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 16:18:39.558278 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.558233    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 16:18:39.558428 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.558403    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 16:18:39.558539 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.558482    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2dwfs\""
Sep 11 16:18:39.558740 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.558720    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 16:18:39.558810 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.558773    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 16:18:39.566188 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.566146    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 16:18:39.574793 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.574769    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-68d6c68898-tstt8"]
Sep 11 16:18:39.681811 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.681424    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506cf99c-d273-4156-b63a-e28179d3270d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.681811 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.681512    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/506cf99c-d273-4156-b63a-e28179d3270d-metrics-client-ca\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.681811 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.681552    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/506cf99c-d273-4156-b63a-e28179d3270d-secret-telemeter-client\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.682580 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.681580    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/506cf99c-d273-4156-b63a-e28179d3270d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.682580 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.682440    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/506cf99c-d273-4156-b63a-e28179d3270d-federate-client-tls\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.682580 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.682467    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thcs4\" (UniqueName: \"kubernetes.io/projected/506cf99c-d273-4156-b63a-e28179d3270d-kube-api-access-thcs4\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.683057 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.682500    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/506cf99c-d273-4156-b63a-e28179d3270d-telemeter-client-tls\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.683057 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.682962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506cf99c-d273-4156-b63a-e28179d3270d-serving-certs-ca-bundle\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.784798 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.784526    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506cf99c-d273-4156-b63a-e28179d3270d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.784798 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.784609    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b08d9859-821d-4f5f-8058-eaf19c990774-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-49rm9\" (UID: \"b08d9859-821d-4f5f-8058-eaf19c990774\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9"
Sep 11 16:18:39.784798 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:18:39.784708    2561 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 11 16:18:39.785078 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:18:39.784891    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b08d9859-821d-4f5f-8058-eaf19c990774-monitoring-plugin-cert podName:b08d9859-821d-4f5f-8058-eaf19c990774 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:40.784752683 +0000 UTC m=+126.124846710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/b08d9859-821d-4f5f-8058-eaf19c990774-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-49rm9" (UID: "b08d9859-821d-4f5f-8058-eaf19c990774") : secret "monitoring-plugin-cert" not found
Sep 11 16:18:39.785235 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.785210    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/506cf99c-d273-4156-b63a-e28179d3270d-metrics-client-ca\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.785368 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.785266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/506cf99c-d273-4156-b63a-e28179d3270d-secret-telemeter-client\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.785368 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.785299    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/506cf99c-d273-4156-b63a-e28179d3270d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.785368 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.785327    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/506cf99c-d273-4156-b63a-e28179d3270d-federate-client-tls\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.785368 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.785351    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thcs4\" (UniqueName: \"kubernetes.io/projected/506cf99c-d273-4156-b63a-e28179d3270d-kube-api-access-thcs4\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.785669 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.785605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506cf99c-d273-4156-b63a-e28179d3270d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.785751 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.785668    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/506cf99c-d273-4156-b63a-e28179d3270d-telemeter-client-tls\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.785751 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.785705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506cf99c-d273-4156-b63a-e28179d3270d-serving-certs-ca-bundle\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.786594 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.786054    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/506cf99c-d273-4156-b63a-e28179d3270d-metrics-client-ca\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.786758 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.786703    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/506cf99c-d273-4156-b63a-e28179d3270d-serving-certs-ca-bundle\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.789427 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.789401    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/506cf99c-d273-4156-b63a-e28179d3270d-telemeter-client-tls\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.789744 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.789722    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/506cf99c-d273-4156-b63a-e28179d3270d-secret-telemeter-client\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.790243 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.790199    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/506cf99c-d273-4156-b63a-e28179d3270d-federate-client-tls\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.791268 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.791244    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/506cf99c-d273-4156-b63a-e28179d3270d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.795070 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.795032    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thcs4\" (UniqueName: \"kubernetes.io/projected/506cf99c-d273-4156-b63a-e28179d3270d-kube-api-access-thcs4\") pod \"telemeter-client-68d6c68898-tstt8\" (UID: \"506cf99c-d273-4156-b63a-e28179d3270d\") " pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:39.871341 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:39.871248    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8"
Sep 11 16:18:40.707140 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.707061    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:18:40.722024 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.720808    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.726296 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.725944    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 16:18:40.726296 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.726044    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 16:18:40.726296 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.726137    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 16:18:40.727142 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.727015    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 16:18:40.727572 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.727554    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 16:18:40.727692 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.727579    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 16:18:40.727766 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.727729    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4gqm6e1v16sc5\""
Sep 11 16:18:40.727815 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.727784    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 16:18:40.727867 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.727630    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 16:18:40.727867 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.727658    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 16:18:40.727965 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.727585    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 16:18:40.727965 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.727914    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 16:18:40.727965 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.727624    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 16:18:40.728292 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.728272    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 16:18:40.728369 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.728278    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7ds8r\""
Sep 11 16:18:40.728431 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.728413    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 16:18:40.730212 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.729937    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 16:18:40.736378 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.736358    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:18:40.794370 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.794324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.794618 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.794585    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqm8f\" (UniqueName: \"kubernetes.io/projected/3d12d629-e6a5-4868-8554-ff0a8318f6e6-kube-api-access-hqm8f\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.794754 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.794740    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.794874 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.794861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.794992 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.794979    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.795138 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.795103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.795345 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.795327    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.795493 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.795477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.795614 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.795601    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.795736 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.795723    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d12d629-e6a5-4868-8554-ff0a8318f6e6-config-out\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.795846 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.795833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-web-config\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.795988 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.795973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.796107 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.796093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.796278 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.796255    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-config\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.796395 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.796381    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.796564 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.796551    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.796765 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.796751    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.796900 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.796887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d12d629-e6a5-4868-8554-ff0a8318f6e6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.797032 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.797016    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.797151 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.797138    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.797802 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.797781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b08d9859-821d-4f5f-8058-eaf19c990774-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-49rm9\" (UID: \"b08d9859-821d-4f5f-8058-eaf19c990774\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9"
Sep 11 16:18:40.800273 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.800239    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b08d9859-821d-4f5f-8058-eaf19c990774-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-49rm9\" (UID: \"b08d9859-821d-4f5f-8058-eaf19c990774\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9"
Sep 11 16:18:40.824236 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.824124    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0340401-505e-41ab-b9c0-fff1f0bc4493","Type":"ContainerStarted","Data":"fe8084d671abe41c32081f7e2e1f0f5fffa3374c301175af9bb07751746321de"}
Sep 11 16:18:40.824236 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.824218    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0340401-505e-41ab-b9c0-fff1f0bc4493","Type":"ContainerStarted","Data":"5e996b81eff51bbd8ee92b70613bd0fa19ba4fd016f9209591cee2efc8a01b23"}
Sep 11 16:18:40.826854 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.826746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k" event={"ID":"b15f23b5-8a1e-4763-838c-ba6550c4918d","Type":"ContainerStarted","Data":"732c0ce6f7e7931683a47d2b4aac1d7e4b54932e1b333668313f8e625219491e"}
Sep 11 16:18:40.826854 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.826827    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k" event={"ID":"b15f23b5-8a1e-4763-838c-ba6550c4918d","Type":"ContainerStarted","Data":"fc9a30397bab59163b3efb4f5a1cc10d431c0852a194d255ea9e156218fdb964"}
Sep 11 16:18:40.826992 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.826869    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k" event={"ID":"b15f23b5-8a1e-4763-838c-ba6550c4918d","Type":"ContainerStarted","Data":"063ad2de163deb91143d049fb0c84fd84d610d549fa4df93e107d89a93517078"}
Sep 11 16:18:40.864659 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.864630    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-68d6c68898-tstt8"]
Sep 11 16:18:40.865803 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.865758    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9"
Sep 11 16:18:40.868765 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:40.868722    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506cf99c_d273_4156_b63a_e28179d3270d.slice/crio-5cc0c2aaeef6026fab7466e9448969e83bd07d4f6a2c88eda9c36ec7cb66d7da WatchSource:0}: Error finding container 5cc0c2aaeef6026fab7466e9448969e83bd07d4f6a2c88eda9c36ec7cb66d7da: Status 404 returned error can't find the container with id 5cc0c2aaeef6026fab7466e9448969e83bd07d4f6a2c88eda9c36ec7cb66d7da
Sep 11 16:18:40.898794 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.898767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-config\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899213 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.898801    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899213 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.898831    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899213 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.898856    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899213 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.898981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d12d629-e6a5-4868-8554-ff0a8318f6e6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899213 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.899015    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899213 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.899041    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899213 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.899074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899213 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.899095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqm8f\" (UniqueName: \"kubernetes.io/projected/3d12d629-e6a5-4868-8554-ff0a8318f6e6-kube-api-access-hqm8f\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899213 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.899122    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899213 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.899148    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899213 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.899189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899725 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.899232    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899725 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.899287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899725 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.899320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899725 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.899353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899725 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.899393    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d12d629-e6a5-4868-8554-ff0a8318f6e6-config-out\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899725 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.899429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-web-config\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899725 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.899472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899725 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.899503    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.899725 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.899685    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.900465 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.900353    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.902804 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.902012    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.902804 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.902279    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-config\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.902804 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.902412    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.902804 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.902690    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.902804 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.902778    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.903620 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.903509    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d12d629-e6a5-4868-8554-ff0a8318f6e6-config-out\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.904043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.903720    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.904043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.903890    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.904043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.903977    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.904919 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.904549    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.904919 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.904560    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.904919 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.904874    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.905326 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.905299    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.905913 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.905884    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.906749 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.906720    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-web-config\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.907084 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.907064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d12d629-e6a5-4868-8554-ff0a8318f6e6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.907744 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.907712    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:40.914283 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:40.914261    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqm8f\" (UniqueName: \"kubernetes.io/projected/3d12d629-e6a5-4868-8554-ff0a8318f6e6-kube-api-access-hqm8f\") pod \"prometheus-k8s-0\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:41.036058 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:41.036012    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:41.151222 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:41.151186    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9"]
Sep 11 16:18:41.155054 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:41.155011    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08d9859_821d_4f5f_8058_eaf19c990774.slice/crio-55f1d873c396a19380b86fd858aa64088238484053f4bb6815297b7711596aa6 WatchSource:0}: Error finding container 55f1d873c396a19380b86fd858aa64088238484053f4bb6815297b7711596aa6: Status 404 returned error can't find the container with id 55f1d873c396a19380b86fd858aa64088238484053f4bb6815297b7711596aa6
Sep 11 16:18:41.325492 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:41.325468    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:18:41.327570 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:41.327546    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d12d629_e6a5_4868_8554_ff0a8318f6e6.slice/crio-9989fe9cef3aadc9f12ef638f4b9b27c266869458744ce68cef72510ca2ab9d7 WatchSource:0}: Error finding container 9989fe9cef3aadc9f12ef638f4b9b27c266869458744ce68cef72510ca2ab9d7: Status 404 returned error can't find the container with id 9989fe9cef3aadc9f12ef638f4b9b27c266869458744ce68cef72510ca2ab9d7
Sep 11 16:18:41.832125 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:41.832092    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8" event={"ID":"506cf99c-d273-4156-b63a-e28179d3270d","Type":"ContainerStarted","Data":"5cc0c2aaeef6026fab7466e9448969e83bd07d4f6a2c88eda9c36ec7cb66d7da"}
Sep 11 16:18:41.835109 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:41.835084    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0340401-505e-41ab-b9c0-fff1f0bc4493","Type":"ContainerStarted","Data":"9eda93326c78d4c065565a192c958e76b1ac70d9742659f3b07a5ad7f4ac75f2"}
Sep 11 16:18:41.835224 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:41.835151    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0340401-505e-41ab-b9c0-fff1f0bc4493","Type":"ContainerStarted","Data":"509be74cc1b0b2dd9f97b180012b369a7573fba5a722dc95aa1eca1d0d1a3d41"}
Sep 11 16:18:41.835224 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:41.835207    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0340401-505e-41ab-b9c0-fff1f0bc4493","Type":"ContainerStarted","Data":"169b45da9fd2b2022bb5f31f61e2e94ef00152afae489a38c31f36bf6f57d13d"}
Sep 11 16:18:41.836141 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:41.836118    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9" event={"ID":"b08d9859-821d-4f5f-8058-eaf19c990774","Type":"ContainerStarted","Data":"55f1d873c396a19380b86fd858aa64088238484053f4bb6815297b7711596aa6"}
Sep 11 16:18:41.837794 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:41.837772    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerID="04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b" exitCode=0
Sep 11 16:18:41.837859 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:41.837834    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d12d629-e6a5-4868-8554-ff0a8318f6e6","Type":"ContainerDied","Data":"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b"}
Sep 11 16:18:41.837907 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:41.837893    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d12d629-e6a5-4868-8554-ff0a8318f6e6","Type":"ContainerStarted","Data":"9989fe9cef3aadc9f12ef638f4b9b27c266869458744ce68cef72510ca2ab9d7"}
Sep 11 16:18:42.848479 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:42.848440    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0340401-505e-41ab-b9c0-fff1f0bc4493","Type":"ContainerStarted","Data":"b722800fad55a579c288129e596d4676d75df01d4fa2e87e1836c46ee326c444"}
Sep 11 16:18:42.852320 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:42.852290    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k" event={"ID":"b15f23b5-8a1e-4763-838c-ba6550c4918d","Type":"ContainerStarted","Data":"503cee9076bf4384fb8ead15362ff381ca8635abedcab358077fef42bf01679a"}
Sep 11 16:18:42.852450 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:42.852361    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k" event={"ID":"b15f23b5-8a1e-4763-838c-ba6550c4918d","Type":"ContainerStarted","Data":"aa4bb5dec15b0e4dcb6ddf365778608b40160e42731713eb2134304317a63f8a"}
Sep 11 16:18:42.852450 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:42.852397    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k" event={"ID":"b15f23b5-8a1e-4763-838c-ba6550c4918d","Type":"ContainerStarted","Data":"c2c41a29f4b8fb31d7054ee5177fe475a98c9e531f79e09e2f39c57f4ea2e2ca"}
Sep 11 16:18:42.852605 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:42.852590    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:42.886480 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:42.886245    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.989739397 podStartE2EDuration="7.886225165s" totalImagesPullingTime="3.88137295s" totalInitContainerRuntime="15.112818ms" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:42.885229346 +0000 UTC m=+128.225323392" watchObservedRunningTime="2026-09-11 16:18:42.886225165 +0000 UTC m=+128.226319212"
Sep 11 16:18:42.916453 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:42.916392    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k" podStartSLOduration=2.508415523 podStartE2EDuration="5.916367618s" totalImagesPullingTime="3.407952095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:42.914042892 +0000 UTC m=+128.254136937" watchObservedRunningTime="2026-09-11 16:18:42.916367618 +0000 UTC m=+128.256461665"
Sep 11 16:18:43.141589 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:43.141503    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-549b9987dc-qhcjq"]
Sep 11 16:18:43.862723 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:43.858268    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8" event={"ID":"506cf99c-d273-4156-b63a-e28179d3270d","Type":"ContainerStarted","Data":"b126a6207fee8a86cd1a688d411d33f8d78e3a180fe6d43d91632c6afbb2fc57"}
Sep 11 16:18:43.862723 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:43.858429    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8" event={"ID":"506cf99c-d273-4156-b63a-e28179d3270d","Type":"ContainerStarted","Data":"e7d69b8f54fa8c1caf987fed7b906137aabba443d172c4ba6d43c32079d2cb79"}
Sep 11 16:18:43.862723 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:43.858482    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8" event={"ID":"506cf99c-d273-4156-b63a-e28179d3270d","Type":"ContainerStarted","Data":"464a84bdad3dee4d18c3bb3c1f7ebf1126ad13a9e5f1b74427e46c4291255510"}
Sep 11 16:18:43.864671 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:43.864631    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9" event={"ID":"b08d9859-821d-4f5f-8058-eaf19c990774","Type":"ContainerStarted","Data":"ca6e23c0ec990ac012074aa8c3c194c028e44e0dd2adbfbde55bf5fa910f93ae"}
Sep 11 16:18:43.865165 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:43.865142    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9"
Sep 11 16:18:43.870222 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:43.870199    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9"
Sep 11 16:18:43.894294 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:43.894246    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-68d6c68898-tstt8" podStartSLOduration=2.60873212 podStartE2EDuration="4.894209811s" totalImagesPullingTime="2.285477691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:43.893325461 +0000 UTC m=+129.233419518" watchObservedRunningTime="2026-09-11 16:18:43.894209811 +0000 UTC m=+129.234303852"
Sep 11 16:18:43.915075 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:43.915020    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-49rm9" podStartSLOduration=2.915855637 podStartE2EDuration="4.915001126s" totalImagesPullingTime="1.999145489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:43.914976696 +0000 UTC m=+129.255070744" watchObservedRunningTime="2026-09-11 16:18:43.915001126 +0000 UTC m=+129.255095171"
Sep 11 16:18:44.945359 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:44.945320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs\") pod \"network-metrics-daemon-z6xtq\" (UID: \"b15c0e16-1077-454c-aafa-16902113243c\") " pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:18:44.950495 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:44.950461    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b15c0e16-1077-454c-aafa-16902113243c-metrics-certs\") pod \"network-metrics-daemon-z6xtq\" (UID: \"b15c0e16-1077-454c-aafa-16902113243c\") " pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:18:45.129983 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:45.129942    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kx52\""
Sep 11 16:18:45.137971 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:45.137941    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z6xtq"
Sep 11 16:18:45.423666 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:45.423519    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z6xtq"]
Sep 11 16:18:45.430060 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:18:45.429901    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15c0e16_1077_454c_aafa_16902113243c.slice/crio-fff68efa9409bf6826c789a6129809a9ca134a41a7d7f5689088a9a7940f9884 WatchSource:0}: Error finding container fff68efa9409bf6826c789a6129809a9ca134a41a7d7f5689088a9a7940f9884: Status 404 returned error can't find the container with id fff68efa9409bf6826c789a6129809a9ca134a41a7d7f5689088a9a7940f9884
Sep 11 16:18:45.879590 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:45.879494    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z6xtq" event={"ID":"b15c0e16-1077-454c-aafa-16902113243c","Type":"ContainerStarted","Data":"fff68efa9409bf6826c789a6129809a9ca134a41a7d7f5689088a9a7940f9884"}
Sep 11 16:18:45.885317 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:45.885275    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d12d629-e6a5-4868-8554-ff0a8318f6e6","Type":"ContainerStarted","Data":"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4"}
Sep 11 16:18:45.885472 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:45.885391    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d12d629-e6a5-4868-8554-ff0a8318f6e6","Type":"ContainerStarted","Data":"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553"}
Sep 11 16:18:45.885472 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:45.885443    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d12d629-e6a5-4868-8554-ff0a8318f6e6","Type":"ContainerStarted","Data":"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60"}
Sep 11 16:18:45.885567 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:45.885486    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d12d629-e6a5-4868-8554-ff0a8318f6e6","Type":"ContainerStarted","Data":"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c"}
Sep 11 16:18:45.885567 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:45.885526    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d12d629-e6a5-4868-8554-ff0a8318f6e6","Type":"ContainerStarted","Data":"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5"}
Sep 11 16:18:45.885567 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:45.885562    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d12d629-e6a5-4868-8554-ff0a8318f6e6","Type":"ContainerStarted","Data":"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde"}
Sep 11 16:18:45.925444 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:45.925383    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.531543983 podStartE2EDuration="5.92535996s" totalImagesPullingTime="3.366998753s" totalInitContainerRuntime="26.817224ms" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:45.92269768 +0000 UTC m=+131.262791732" watchObservedRunningTime="2026-09-11 16:18:45.92535996 +0000 UTC m=+131.265454006"
Sep 11 16:18:46.037233 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:46.037194    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:18:46.891251 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:46.891210    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z6xtq" event={"ID":"b15c0e16-1077-454c-aafa-16902113243c","Type":"ContainerStarted","Data":"39045bfe8b9703f575857f730c5420f927cb0d0e0ee1349da77891da6f27db6f"}
Sep 11 16:18:46.891415 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:46.891274    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z6xtq" event={"ID":"b15c0e16-1077-454c-aafa-16902113243c","Type":"ContainerStarted","Data":"bab5f7ff1a2dfed6b9259c2442af84006049ff9a0dcf95a68a6b50d15918285a"}
Sep 11 16:18:48.872034 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:48.872005    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6c75b4f8f-xqr2k"
Sep 11 16:18:48.905193 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:18:48.903746    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-z6xtq" podStartSLOduration=133.026975785 podStartE2EDuration="2m13.9037214s" totalImagesPullingTime="876.745615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:46.922918215 +0000 UTC m=+132.263012260" watchObservedRunningTime="2026-09-11 16:18:48.9037214 +0000 UTC m=+134.243815446"
Sep 11 16:19:08.164220 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.164130    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-549b9987dc-qhcjq" podUID="da348161-6dda-45e7-b4a4-4a4948eeb2a7" containerName="registry" containerID="cri-o://5addadf6c7ab20a49a834d836dbd7f3623b303bb76f3694e623352d39551fd83" gracePeriod=30
Sep 11 16:19:08.404794 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.404770    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:19:08.566456 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.566368    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da348161-6dda-45e7-b4a4-4a4948eeb2a7-installation-pull-secrets\") pod \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") "
Sep 11 16:19:08.566456 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.566403    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-certificates\") pod \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") "
Sep 11 16:19:08.566456 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.566422    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da348161-6dda-45e7-b4a4-4a4948eeb2a7-trusted-ca\") pod \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") "
Sep 11 16:19:08.566703 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.566470    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da348161-6dda-45e7-b4a4-4a4948eeb2a7-image-registry-private-configuration\") pod \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") "
Sep 11 16:19:08.566703 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.566522    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls\") pod \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") "
Sep 11 16:19:08.566703 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.566545    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktttd\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-kube-api-access-ktttd\") pod \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") "
Sep 11 16:19:08.566703 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.566586    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-bound-sa-token\") pod \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") "
Sep 11 16:19:08.566865 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.566713    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da348161-6dda-45e7-b4a4-4a4948eeb2a7-ca-trust-extracted\") pod \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\" (UID: \"da348161-6dda-45e7-b4a4-4a4948eeb2a7\") "
Sep 11 16:19:08.566919 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.566888    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da348161-6dda-45e7-b4a4-4a4948eeb2a7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "da348161-6dda-45e7-b4a4-4a4948eeb2a7" (UID: "da348161-6dda-45e7-b4a4-4a4948eeb2a7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:08.566978 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.566909    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "da348161-6dda-45e7-b4a4-4a4948eeb2a7" (UID: "da348161-6dda-45e7-b4a4-4a4948eeb2a7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:08.567252 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.567046    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-certificates\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:08.567252 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.567236    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da348161-6dda-45e7-b4a4-4a4948eeb2a7-trusted-ca\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:08.569137 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.569105    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da348161-6dda-45e7-b4a4-4a4948eeb2a7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "da348161-6dda-45e7-b4a4-4a4948eeb2a7" (UID: "da348161-6dda-45e7-b4a4-4a4948eeb2a7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:08.569337 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.569107    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "da348161-6dda-45e7-b4a4-4a4948eeb2a7" (UID: "da348161-6dda-45e7-b4a4-4a4948eeb2a7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:19:08.569408 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.569353    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da348161-6dda-45e7-b4a4-4a4948eeb2a7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "da348161-6dda-45e7-b4a4-4a4948eeb2a7" (UID: "da348161-6dda-45e7-b4a4-4a4948eeb2a7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:08.569463 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.569408    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-kube-api-access-ktttd" (OuterVolumeSpecName: "kube-api-access-ktttd") pod "da348161-6dda-45e7-b4a4-4a4948eeb2a7" (UID: "da348161-6dda-45e7-b4a4-4a4948eeb2a7"). InnerVolumeSpecName "kube-api-access-ktttd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:19:08.569463 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.569456    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "da348161-6dda-45e7-b4a4-4a4948eeb2a7" (UID: "da348161-6dda-45e7-b4a4-4a4948eeb2a7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:19:08.574864 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.574838    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da348161-6dda-45e7-b4a4-4a4948eeb2a7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "da348161-6dda-45e7-b4a4-4a4948eeb2a7" (UID: "da348161-6dda-45e7-b4a4-4a4948eeb2a7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:08.668227 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.668192    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-bound-sa-token\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:08.668227 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.668219    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da348161-6dda-45e7-b4a4-4a4948eeb2a7-ca-trust-extracted\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:08.668227 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.668229    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da348161-6dda-45e7-b4a4-4a4948eeb2a7-installation-pull-secrets\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:08.668443 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.668241    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da348161-6dda-45e7-b4a4-4a4948eeb2a7-image-registry-private-configuration\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:08.668443 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.668249    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-registry-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:08.668443 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:08.668257    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktttd\" (UniqueName: \"kubernetes.io/projected/da348161-6dda-45e7-b4a4-4a4948eeb2a7-kube-api-access-ktttd\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:09.006557 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:09.006519    2561 generic.go:356] "Generic (PLEG): container finished" podID="da348161-6dda-45e7-b4a4-4a4948eeb2a7" containerID="5addadf6c7ab20a49a834d836dbd7f3623b303bb76f3694e623352d39551fd83" exitCode=0
Sep 11 16:19:09.006716 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:09.006567    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-549b9987dc-qhcjq"
Sep 11 16:19:09.006716 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:09.006606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-549b9987dc-qhcjq" event={"ID":"da348161-6dda-45e7-b4a4-4a4948eeb2a7","Type":"ContainerDied","Data":"5addadf6c7ab20a49a834d836dbd7f3623b303bb76f3694e623352d39551fd83"}
Sep 11 16:19:09.006716 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:09.006683    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-549b9987dc-qhcjq" event={"ID":"da348161-6dda-45e7-b4a4-4a4948eeb2a7","Type":"ContainerDied","Data":"5738207769b75728e90202cdd07a887312dd4555b8d7c82675ce6ec3873ed236"}
Sep 11 16:19:09.006716 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:09.006713    2561 scope.go:117] "RemoveContainer" containerID="5addadf6c7ab20a49a834d836dbd7f3623b303bb76f3694e623352d39551fd83"
Sep 11 16:19:09.015441 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:09.015424    2561 scope.go:117] "RemoveContainer" containerID="5addadf6c7ab20a49a834d836dbd7f3623b303bb76f3694e623352d39551fd83"
Sep 11 16:19:09.015718 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:19:09.015700    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5addadf6c7ab20a49a834d836dbd7f3623b303bb76f3694e623352d39551fd83\": container with ID starting with 5addadf6c7ab20a49a834d836dbd7f3623b303bb76f3694e623352d39551fd83 not found: ID does not exist" containerID="5addadf6c7ab20a49a834d836dbd7f3623b303bb76f3694e623352d39551fd83"
Sep 11 16:19:09.015790 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:09.015724    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5addadf6c7ab20a49a834d836dbd7f3623b303bb76f3694e623352d39551fd83"} err="failed to get container status \"5addadf6c7ab20a49a834d836dbd7f3623b303bb76f3694e623352d39551fd83\": rpc error: code = NotFound desc = could not find container \"5addadf6c7ab20a49a834d836dbd7f3623b303bb76f3694e623352d39551fd83\": container with ID starting with 5addadf6c7ab20a49a834d836dbd7f3623b303bb76f3694e623352d39551fd83 not found: ID does not exist"
Sep 11 16:19:09.037217 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:09.037170    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-549b9987dc-qhcjq"]
Sep 11 16:19:09.040427 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:09.040404    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-549b9987dc-qhcjq"]
Sep 11 16:19:09.322106 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:09.322026    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da348161-6dda-45e7-b4a4-4a4948eeb2a7" path="/var/lib/kubelet/pods/da348161-6dda-45e7-b4a4-4a4948eeb2a7/volumes"
Sep 11 16:19:41.036702 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:41.036663    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.054947 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:41.054919    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:41.168858 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:41.168835    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:54.981169 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:54.981136    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:19:54.981570 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:54.981416    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="alertmanager" containerID="cri-o://5e996b81eff51bbd8ee92b70613bd0fa19ba4fd016f9209591cee2efc8a01b23" gracePeriod=120
Sep 11 16:19:54.981570 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:54.981491    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="kube-rbac-proxy-metric" containerID="cri-o://9eda93326c78d4c065565a192c958e76b1ac70d9742659f3b07a5ad7f4ac75f2" gracePeriod=120
Sep 11 16:19:54.981570 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:54.981532    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="kube-rbac-proxy-web" containerID="cri-o://169b45da9fd2b2022bb5f31f61e2e94ef00152afae489a38c31f36bf6f57d13d" gracePeriod=120
Sep 11 16:19:54.981735 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:54.981569    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="kube-rbac-proxy" containerID="cri-o://509be74cc1b0b2dd9f97b180012b369a7573fba5a722dc95aa1eca1d0d1a3d41" gracePeriod=120
Sep 11 16:19:54.981735 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:54.981597    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="config-reloader" containerID="cri-o://fe8084d671abe41c32081f7e2e1f0f5fffa3374c301175af9bb07751746321de" gracePeriod=120
Sep 11 16:19:54.981735 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:54.981611    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="prom-label-proxy" containerID="cri-o://b722800fad55a579c288129e596d4676d75df01d4fa2e87e1836c46ee326c444" gracePeriod=120
Sep 11 16:19:55.214030 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:55.213995    2561 generic.go:356] "Generic (PLEG): container finished" podID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerID="b722800fad55a579c288129e596d4676d75df01d4fa2e87e1836c46ee326c444" exitCode=0
Sep 11 16:19:55.214030 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:55.214021    2561 generic.go:356] "Generic (PLEG): container finished" podID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerID="509be74cc1b0b2dd9f97b180012b369a7573fba5a722dc95aa1eca1d0d1a3d41" exitCode=0
Sep 11 16:19:55.214030 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:55.214027    2561 generic.go:356] "Generic (PLEG): container finished" podID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerID="fe8084d671abe41c32081f7e2e1f0f5fffa3374c301175af9bb07751746321de" exitCode=0
Sep 11 16:19:55.214030 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:55.214033    2561 generic.go:356] "Generic (PLEG): container finished" podID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerID="5e996b81eff51bbd8ee92b70613bd0fa19ba4fd016f9209591cee2efc8a01b23" exitCode=0
Sep 11 16:19:55.214300 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:55.214062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0340401-505e-41ab-b9c0-fff1f0bc4493","Type":"ContainerDied","Data":"b722800fad55a579c288129e596d4676d75df01d4fa2e87e1836c46ee326c444"}
Sep 11 16:19:55.214300 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:55.214123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0340401-505e-41ab-b9c0-fff1f0bc4493","Type":"ContainerDied","Data":"509be74cc1b0b2dd9f97b180012b369a7573fba5a722dc95aa1eca1d0d1a3d41"}
Sep 11 16:19:55.214300 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:55.214149    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0340401-505e-41ab-b9c0-fff1f0bc4493","Type":"ContainerDied","Data":"fe8084d671abe41c32081f7e2e1f0f5fffa3374c301175af9bb07751746321de"}
Sep 11 16:19:55.214300 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:55.214191    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0340401-505e-41ab-b9c0-fff1f0bc4493","Type":"ContainerDied","Data":"5e996b81eff51bbd8ee92b70613bd0fa19ba4fd016f9209591cee2efc8a01b23"}
Sep 11 16:19:56.221283 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.221252    2561 generic.go:356] "Generic (PLEG): container finished" podID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerID="9eda93326c78d4c065565a192c958e76b1ac70d9742659f3b07a5ad7f4ac75f2" exitCode=0
Sep 11 16:19:56.221283 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.221278    2561 generic.go:356] "Generic (PLEG): container finished" podID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerID="169b45da9fd2b2022bb5f31f61e2e94ef00152afae489a38c31f36bf6f57d13d" exitCode=0
Sep 11 16:19:56.221619 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.221322    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0340401-505e-41ab-b9c0-fff1f0bc4493","Type":"ContainerDied","Data":"9eda93326c78d4c065565a192c958e76b1ac70d9742659f3b07a5ad7f4ac75f2"}
Sep 11 16:19:56.221619 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.221383    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0340401-505e-41ab-b9c0-fff1f0bc4493","Type":"ContainerDied","Data":"169b45da9fd2b2022bb5f31f61e2e94ef00152afae489a38c31f36bf6f57d13d"}
Sep 11 16:19:56.221619 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.221410    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0340401-505e-41ab-b9c0-fff1f0bc4493","Type":"ContainerDied","Data":"7ee7a9493fbdb86c5e96b617ac8f423b193a2faaf686c8db00fbc527cf2f5709"}
Sep 11 16:19:56.221619 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.221436    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ee7a9493fbdb86c5e96b617ac8f423b193a2faaf686c8db00fbc527cf2f5709"
Sep 11 16:19:56.221937 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.221912    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:56.368317 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.368221    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy\") pod \"c0340401-505e-41ab-b9c0-fff1f0bc4493\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") "
Sep 11 16:19:56.368317 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.368259    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0340401-505e-41ab-b9c0-fff1f0bc4493-metrics-client-ca\") pod \"c0340401-505e-41ab-b9c0-fff1f0bc4493\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") "
Sep 11 16:19:56.368317 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.368285    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0340401-505e-41ab-b9c0-fff1f0bc4493-alertmanager-trusted-ca-bundle\") pod \"c0340401-505e-41ab-b9c0-fff1f0bc4493\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") "
Sep 11 16:19:56.368574 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.368350    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0340401-505e-41ab-b9c0-fff1f0bc4493-alertmanager-main-db\") pod \"c0340401-505e-41ab-b9c0-fff1f0bc4493\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") "
Sep 11 16:19:56.368574 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.368382    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0340401-505e-41ab-b9c0-fff1f0bc4493-config-out\") pod \"c0340401-505e-41ab-b9c0-fff1f0bc4493\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") "
Sep 11 16:19:56.368574 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.368404    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c0340401-505e-41ab-b9c0-fff1f0bc4493\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") "
Sep 11 16:19:56.368720 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.368659    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy-web\") pod \"c0340401-505e-41ab-b9c0-fff1f0bc4493\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") "
Sep 11 16:19:56.368720 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.368707    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-config-volume\") pod \"c0340401-505e-41ab-b9c0-fff1f0bc4493\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") "
Sep 11 16:19:56.368820 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.368709    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0340401-505e-41ab-b9c0-fff1f0bc4493-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c0340401-505e-41ab-b9c0-fff1f0bc4493" (UID: "c0340401-505e-41ab-b9c0-fff1f0bc4493"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:56.368820 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.368750    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25czd\" (UniqueName: \"kubernetes.io/projected/c0340401-505e-41ab-b9c0-fff1f0bc4493-kube-api-access-25czd\") pod \"c0340401-505e-41ab-b9c0-fff1f0bc4493\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") "
Sep 11 16:19:56.368820 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.368762    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0340401-505e-41ab-b9c0-fff1f0bc4493-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c0340401-505e-41ab-b9c0-fff1f0bc4493" (UID: "c0340401-505e-41ab-b9c0-fff1f0bc4493"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:56.368820 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.368780    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-cluster-tls-config\") pod \"c0340401-505e-41ab-b9c0-fff1f0bc4493\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") "
Sep 11 16:19:56.368820 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.368808    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0340401-505e-41ab-b9c0-fff1f0bc4493-tls-assets\") pod \"c0340401-505e-41ab-b9c0-fff1f0bc4493\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") "
Sep 11 16:19:56.369028 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.368820    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0340401-505e-41ab-b9c0-fff1f0bc4493-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c0340401-505e-41ab-b9c0-fff1f0bc4493" (UID: "c0340401-505e-41ab-b9c0-fff1f0bc4493"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:56.369028 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.368835    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-main-tls\") pod \"c0340401-505e-41ab-b9c0-fff1f0bc4493\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") "
Sep 11 16:19:56.369028 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.368871    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-web-config\") pod \"c0340401-505e-41ab-b9c0-fff1f0bc4493\" (UID: \"c0340401-505e-41ab-b9c0-fff1f0bc4493\") "
Sep 11 16:19:56.369197 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.369159    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0340401-505e-41ab-b9c0-fff1f0bc4493-metrics-client-ca\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:56.369259 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.369211    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0340401-505e-41ab-b9c0-fff1f0bc4493-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:56.369259 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.369226    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0340401-505e-41ab-b9c0-fff1f0bc4493-alertmanager-main-db\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:56.371445 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.371401    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c0340401-505e-41ab-b9c0-fff1f0bc4493" (UID: "c0340401-505e-41ab-b9c0-fff1f0bc4493"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:56.371569 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.371472    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0340401-505e-41ab-b9c0-fff1f0bc4493-config-out" (OuterVolumeSpecName: "config-out") pod "c0340401-505e-41ab-b9c0-fff1f0bc4493" (UID: "c0340401-505e-41ab-b9c0-fff1f0bc4493"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:56.371619 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.371575    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c0340401-505e-41ab-b9c0-fff1f0bc4493" (UID: "c0340401-505e-41ab-b9c0-fff1f0bc4493"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:56.371757 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.371720    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c0340401-505e-41ab-b9c0-fff1f0bc4493" (UID: "c0340401-505e-41ab-b9c0-fff1f0bc4493"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:56.372325 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.372274    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0340401-505e-41ab-b9c0-fff1f0bc4493-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c0340401-505e-41ab-b9c0-fff1f0bc4493" (UID: "c0340401-505e-41ab-b9c0-fff1f0bc4493"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:19:56.372325 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.372316    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-config-volume" (OuterVolumeSpecName: "config-volume") pod "c0340401-505e-41ab-b9c0-fff1f0bc4493" (UID: "c0340401-505e-41ab-b9c0-fff1f0bc4493"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:56.373048 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.373025    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0340401-505e-41ab-b9c0-fff1f0bc4493-kube-api-access-25czd" (OuterVolumeSpecName: "kube-api-access-25czd") pod "c0340401-505e-41ab-b9c0-fff1f0bc4493" (UID: "c0340401-505e-41ab-b9c0-fff1f0bc4493"). InnerVolumeSpecName "kube-api-access-25czd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:19:56.373165 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.373147    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c0340401-505e-41ab-b9c0-fff1f0bc4493" (UID: "c0340401-505e-41ab-b9c0-fff1f0bc4493"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:56.376362 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.376329    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c0340401-505e-41ab-b9c0-fff1f0bc4493" (UID: "c0340401-505e-41ab-b9c0-fff1f0bc4493"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:56.383221 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.383191    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-web-config" (OuterVolumeSpecName: "web-config") pod "c0340401-505e-41ab-b9c0-fff1f0bc4493" (UID: "c0340401-505e-41ab-b9c0-fff1f0bc4493"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:56.469912 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.469872    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:56.469912 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.469909    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0340401-505e-41ab-b9c0-fff1f0bc4493-config-out\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:56.469912 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.469918    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:56.469912 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.469927    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:56.470167 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.469937    2561 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-config-volume\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:56.470167 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.469945    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25czd\" (UniqueName: \"kubernetes.io/projected/c0340401-505e-41ab-b9c0-fff1f0bc4493-kube-api-access-25czd\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:56.470167 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.469955    2561 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-cluster-tls-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:56.470167 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.469963    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0340401-505e-41ab-b9c0-fff1f0bc4493-tls-assets\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:56.470167 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.469971    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-secret-alertmanager-main-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:56.470167 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:56.469979    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0340401-505e-41ab-b9c0-fff1f0bc4493-web-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:57.225502 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.225468    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.263066 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.263035    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:19:57.268383 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.268354    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:19:57.321113 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.321024    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" path="/var/lib/kubelet/pods/c0340401-505e-41ab-b9c0-fff1f0bc4493/volumes"
Sep 11 16:19:57.324572 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324546    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:19:57.324856 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324845    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="config-reloader"
Sep 11 16:19:57.324896 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324858    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="config-reloader"
Sep 11 16:19:57.324896 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324869    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="kube-rbac-proxy"
Sep 11 16:19:57.324896 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324875    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="kube-rbac-proxy"
Sep 11 16:19:57.324896 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324882    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="alertmanager"
Sep 11 16:19:57.324896 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324887    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="alertmanager"
Sep 11 16:19:57.325043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324900    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="kube-rbac-proxy-web"
Sep 11 16:19:57.325043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324906    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="kube-rbac-proxy-web"
Sep 11 16:19:57.325043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324913    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="prom-label-proxy"
Sep 11 16:19:57.325043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324919    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="prom-label-proxy"
Sep 11 16:19:57.325043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324925    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="kube-rbac-proxy-metric"
Sep 11 16:19:57.325043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324930    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="kube-rbac-proxy-metric"
Sep 11 16:19:57.325043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324939    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="init-config-reloader"
Sep 11 16:19:57.325043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324943    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="init-config-reloader"
Sep 11 16:19:57.325043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324954    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da348161-6dda-45e7-b4a4-4a4948eeb2a7" containerName="registry"
Sep 11 16:19:57.325043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.324959    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="da348161-6dda-45e7-b4a4-4a4948eeb2a7" containerName="registry"
Sep 11 16:19:57.325043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.325016    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="kube-rbac-proxy-web"
Sep 11 16:19:57.325043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.325027    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="alertmanager"
Sep 11 16:19:57.325043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.325033    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="prom-label-proxy"
Sep 11 16:19:57.325043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.325039    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="config-reloader"
Sep 11 16:19:57.325043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.325045    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="kube-rbac-proxy"
Sep 11 16:19:57.325512 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.325051    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0340401-505e-41ab-b9c0-fff1f0bc4493" containerName="kube-rbac-proxy-metric"
Sep 11 16:19:57.325512 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.325056    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="da348161-6dda-45e7-b4a4-4a4948eeb2a7" containerName="registry"
Sep 11 16:19:57.329914 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.329896    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.340293 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.340271    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 16:19:57.340500 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.340487    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 16:19:57.340646 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.340614    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 16:19:57.340728 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.340659    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 16:19:57.340728 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.340717    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 16:19:57.340867 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.340778    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 16:19:57.340867 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.340799    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 16:19:57.344083 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.344066    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zb8g5\""
Sep 11 16:19:57.346693 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.346679    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 16:19:57.366372 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.366348    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 16:19:57.367743 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.367713    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:19:57.479438 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.479406    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.479613 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.479446    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5eed059-2d45-48e6-a379-3941236f35fd-config-out\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.479613 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.479466    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.479613 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.479569    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.479851 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.479611    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.479851 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.479641    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2bm4\" (UniqueName: \"kubernetes.io/projected/b5eed059-2d45-48e6-a379-3941236f35fd-kube-api-access-n2bm4\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.479851 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.479687    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.479851 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.479716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5eed059-2d45-48e6-a379-3941236f35fd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.479851 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.479745    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5eed059-2d45-48e6-a379-3941236f35fd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.479851 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.479787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.480040 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.479858    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5eed059-2d45-48e6-a379-3941236f35fd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.480040 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.479888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-web-config\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.480040 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.479905    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5eed059-2d45-48e6-a379-3941236f35fd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.580563 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.580483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5eed059-2d45-48e6-a379-3941236f35fd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.582002 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.580919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.582002 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.581093    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5eed059-2d45-48e6-a379-3941236f35fd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.582002 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.581150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-web-config\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.582002 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.581210    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5eed059-2d45-48e6-a379-3941236f35fd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.582002 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.581266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.582002 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.581312    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5eed059-2d45-48e6-a379-3941236f35fd-config-out\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.582002 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.581346    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.582002 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.581393    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.582002 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.581427    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.582002 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.581460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2bm4\" (UniqueName: \"kubernetes.io/projected/b5eed059-2d45-48e6-a379-3941236f35fd-kube-api-access-n2bm4\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.582002 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.581507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.582002 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.581550    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5eed059-2d45-48e6-a379-3941236f35fd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.583092 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.583061    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5eed059-2d45-48e6-a379-3941236f35fd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.584029 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.583995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.585850 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.585163    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5eed059-2d45-48e6-a379-3941236f35fd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.586267 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.586201    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5eed059-2d45-48e6-a379-3941236f35fd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.588148 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.588120    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.588890 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.588759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-web-config\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.588890 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.588781    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5eed059-2d45-48e6-a379-3941236f35fd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.589674 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.589606    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5eed059-2d45-48e6-a379-3941236f35fd-config-out\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.589674 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.589629    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.589789 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.589734    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.590901 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.590879    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.590982 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.590933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5eed059-2d45-48e6-a379-3941236f35fd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.597931 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.597908    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2bm4\" (UniqueName: \"kubernetes.io/projected/b5eed059-2d45-48e6-a379-3941236f35fd-kube-api-access-n2bm4\") pod \"alertmanager-main-0\" (UID: \"b5eed059-2d45-48e6-a379-3941236f35fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.639832 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.639800    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:57.943580 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:57.943536    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:19:57.947896 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:19:57.947862    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5eed059_2d45_48e6_a379_3941236f35fd.slice/crio-068069133eab2f20a7ea4618f7bc9fed9c2572cd0184c78608f7cd94e4635700 WatchSource:0}: Error finding container 068069133eab2f20a7ea4618f7bc9fed9c2572cd0184c78608f7cd94e4635700: Status 404 returned error can't find the container with id 068069133eab2f20a7ea4618f7bc9fed9c2572cd0184c78608f7cd94e4635700
Sep 11 16:19:58.232012 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:58.231917    2561 generic.go:356] "Generic (PLEG): container finished" podID="b5eed059-2d45-48e6-a379-3941236f35fd" containerID="69f438f92c76c45aa1843277190d0114904e7408bf9ca0901090cca8e70cab73" exitCode=0
Sep 11 16:19:58.232407 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:58.232007    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5eed059-2d45-48e6-a379-3941236f35fd","Type":"ContainerDied","Data":"69f438f92c76c45aa1843277190d0114904e7408bf9ca0901090cca8e70cab73"}
Sep 11 16:19:58.232407 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:58.232094    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5eed059-2d45-48e6-a379-3941236f35fd","Type":"ContainerStarted","Data":"068069133eab2f20a7ea4618f7bc9fed9c2572cd0184c78608f7cd94e4635700"}
Sep 11 16:19:59.240883 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.240840    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5eed059-2d45-48e6-a379-3941236f35fd","Type":"ContainerStarted","Data":"9ce783fba8660c8effaf6cdad38b5911237a26e820b4cb4fc5fc98b23e831870"}
Sep 11 16:19:59.241293 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.240961    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5eed059-2d45-48e6-a379-3941236f35fd","Type":"ContainerStarted","Data":"bbf820ef711ab09e82dcbc57e4acf03b4363b3c085971f7469efb3d25220bf4b"}
Sep 11 16:19:59.241293 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.241037    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5eed059-2d45-48e6-a379-3941236f35fd","Type":"ContainerStarted","Data":"1804efc7aecb4dd5df268f92149a47c1976451d8c6e8c7f11baebcfe371f6360"}
Sep 11 16:19:59.241293 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.241083    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5eed059-2d45-48e6-a379-3941236f35fd","Type":"ContainerStarted","Data":"24478a74ae4b829d6a4cda9792f9ec186cdc849a4e557efd1b69ae4acc5708cb"}
Sep 11 16:19:59.241293 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.241127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5eed059-2d45-48e6-a379-3941236f35fd","Type":"ContainerStarted","Data":"210c64add7da35aa76dca7f03eeea66d5461cdb46e477694f6e1f49fcce61dd5"}
Sep 11 16:19:59.241293 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.241210    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5eed059-2d45-48e6-a379-3941236f35fd","Type":"ContainerStarted","Data":"4492996d5da900dad19aa577fc1ca4c8791d98fe1d0a406d68a4ea15f2b17ec6"}
Sep 11 16:19:59.279540 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.279468    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.262378636 podStartE2EDuration="2.279446195s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.067559ms" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:59.276684104 +0000 UTC m=+204.616778151" watchObservedRunningTime="2026-09-11 16:19:59.279446195 +0000 UTC m=+204.619540239"
Sep 11 16:19:59.301397 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.301357    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:19:59.302581 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.302544    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="prometheus" containerID="cri-o://91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde" gracePeriod=600
Sep 11 16:19:59.303094 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.302913    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="kube-rbac-proxy" containerID="cri-o://0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553" gracePeriod=600
Sep 11 16:19:59.303094 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.302952    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="thanos-sidecar" containerID="cri-o://73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c" gracePeriod=600
Sep 11 16:19:59.303094 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.302982    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="config-reloader" containerID="cri-o://dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5" gracePeriod=600
Sep 11 16:19:59.303094 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.303053    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="kube-rbac-proxy-thanos" containerID="cri-o://62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4" gracePeriod=600
Sep 11 16:19:59.303420 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.302920    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="kube-rbac-proxy-web" containerID="cri-o://a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60" gracePeriod=600
Sep 11 16:19:59.557236 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.557207    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:19:59.704114 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704077    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-config\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.704270 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704124    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-trusted-ca-bundle\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.704270 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704168    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-web-config\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.704368 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704265    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-1\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.704368 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704299    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-metrics-client-ca\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.704368 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704330    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-thanos-prometheus-http-client-file\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.704514 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704485    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-kubelet-serving-ca-bundle\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.704574 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704537    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-serving-certs-ca-bundle\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.704625 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704567    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.704625 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704569    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:59.704625 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704581    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:59.704625 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704594    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-metrics-client-certs\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.704818 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704650    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqm8f\" (UniqueName: \"kubernetes.io/projected/3d12d629-e6a5-4868-8554-ff0a8318f6e6-kube-api-access-hqm8f\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.704818 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704670    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:59.704818 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704681    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-grpc-tls\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.704818 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704743    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-0\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.704818 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704771    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d12d629-e6a5-4868-8554-ff0a8318f6e6-config-out\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.704818 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704797    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d12d629-e6a5-4868-8554-ff0a8318f6e6-tls-assets\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.705103 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704829    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-tls\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.705103 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704844    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:59.705103 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704854    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-kube-rbac-proxy\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.705103 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704894    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-db\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.705103 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704934    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.705103 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.704982    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-2\") pod \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\" (UID: \"3d12d629-e6a5-4868-8554-ff0a8318f6e6\") "
Sep 11 16:19:59.705452 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.705311    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.705452 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.705333    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.705452 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.705349    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-metrics-client-ca\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.705452 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.705369    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.706383 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.705651    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:59.706852 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.706801    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:59.706852 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.706804    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-config" (OuterVolumeSpecName: "config") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:59.707010 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.706939    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:59.707367 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.707127    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:59.707367 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.707290    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:59.707367 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.707349    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:59.708368 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.708338    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:19:59.708762 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.708695    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:59.708861 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.708806    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:59.709217 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.709165    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d12d629-e6a5-4868-8554-ff0a8318f6e6-config-out" (OuterVolumeSpecName: "config-out") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:19:59.709300 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.709248    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d12d629-e6a5-4868-8554-ff0a8318f6e6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:19:59.709351 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.709335    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:59.709435 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.709421    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:59.710163 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.710144    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d12d629-e6a5-4868-8554-ff0a8318f6e6-kube-api-access-hqm8f" (OuterVolumeSpecName: "kube-api-access-hqm8f") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "kube-api-access-hqm8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:19:59.717916 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.717893    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-web-config" (OuterVolumeSpecName: "web-config") pod "3d12d629-e6a5-4868-8554-ff0a8318f6e6" (UID: "3d12d629-e6a5-4868-8554-ff0a8318f6e6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:19:59.805963 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.805873    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqm8f\" (UniqueName: \"kubernetes.io/projected/3d12d629-e6a5-4868-8554-ff0a8318f6e6-kube-api-access-hqm8f\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.805963 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.805904    2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-grpc-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.805963 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.805923    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.805963 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.805938    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d12d629-e6a5-4868-8554-ff0a8318f6e6-config-out\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.805963 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.805952    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d12d629-e6a5-4868-8554-ff0a8318f6e6-tls-assets\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.805963 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.805964    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.806298 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.805976    2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-kube-rbac-proxy\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.806298 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.805988    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-db\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.806298 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.806002    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.806298 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.806015    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.806298 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.806026    2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.806298 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.806039    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-web-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.806298 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.806050    2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.806298 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.806063    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d12d629-e6a5-4868-8554-ff0a8318f6e6-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.806298 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.806079    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:19:59.806298 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:19:59.806092    2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d12d629-e6a5-4868-8554-ff0a8318f6e6-secret-metrics-client-certs\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:20:00.253705 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.253670    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerID="62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4" exitCode=0
Sep 11 16:20:00.253705 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.253695    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerID="0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553" exitCode=0
Sep 11 16:20:00.253705 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.253703    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerID="a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60" exitCode=0
Sep 11 16:20:00.253705 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.253709    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerID="73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c" exitCode=0
Sep 11 16:20:00.253705 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.253715    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerID="dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5" exitCode=0
Sep 11 16:20:00.254287 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.253721    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerID="91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde" exitCode=0
Sep 11 16:20:00.254287 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.253756    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.254287 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.253753    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d12d629-e6a5-4868-8554-ff0a8318f6e6","Type":"ContainerDied","Data":"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4"}
Sep 11 16:20:00.254287 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.253865    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d12d629-e6a5-4868-8554-ff0a8318f6e6","Type":"ContainerDied","Data":"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553"}
Sep 11 16:20:00.254287 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.253909    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d12d629-e6a5-4868-8554-ff0a8318f6e6","Type":"ContainerDied","Data":"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60"}
Sep 11 16:20:00.254287 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.253929    2561 scope.go:117] "RemoveContainer" containerID="62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4"
Sep 11 16:20:00.254287 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.253951    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d12d629-e6a5-4868-8554-ff0a8318f6e6","Type":"ContainerDied","Data":"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c"}
Sep 11 16:20:00.254287 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.253991    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d12d629-e6a5-4868-8554-ff0a8318f6e6","Type":"ContainerDied","Data":"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5"}
Sep 11 16:20:00.254287 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.254029    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d12d629-e6a5-4868-8554-ff0a8318f6e6","Type":"ContainerDied","Data":"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde"}
Sep 11 16:20:00.254287 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.254069    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d12d629-e6a5-4868-8554-ff0a8318f6e6","Type":"ContainerDied","Data":"9989fe9cef3aadc9f12ef638f4b9b27c266869458744ce68cef72510ca2ab9d7"}
Sep 11 16:20:00.264759 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.264740    2561 scope.go:117] "RemoveContainer" containerID="0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553"
Sep 11 16:20:00.272612 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.272591    2561 scope.go:117] "RemoveContainer" containerID="a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60"
Sep 11 16:20:00.279419 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.279394    2561 scope.go:117] "RemoveContainer" containerID="73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c"
Sep 11 16:20:00.286503 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.286486    2561 scope.go:117] "RemoveContainer" containerID="dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5"
Sep 11 16:20:00.290337 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.290313    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:20:00.294326 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.294306    2561 scope.go:117] "RemoveContainer" containerID="91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde"
Sep 11 16:20:00.301678 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.301658    2561 scope.go:117] "RemoveContainer" containerID="04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b"
Sep 11 16:20:00.308872 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.308714    2561 scope.go:117] "RemoveContainer" containerID="62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4"
Sep 11 16:20:00.308994 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:20:00.308977    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4\": container with ID starting with 62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4 not found: ID does not exist" containerID="62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4"
Sep 11 16:20:00.309032 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.309004    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4"} err="failed to get container status \"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4\": rpc error: code = NotFound desc = could not find container \"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4\": container with ID starting with 62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4 not found: ID does not exist"
Sep 11 16:20:00.309032 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.309022    2561 scope.go:117] "RemoveContainer" containerID="0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553"
Sep 11 16:20:00.309288 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:20:00.309263    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553\": container with ID starting with 0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553 not found: ID does not exist" containerID="0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553"
Sep 11 16:20:00.309366 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.309289    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553"} err="failed to get container status \"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553\": rpc error: code = NotFound desc = could not find container \"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553\": container with ID starting with 0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553 not found: ID does not exist"
Sep 11 16:20:00.309366 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.309305    2561 scope.go:117] "RemoveContainer" containerID="a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60"
Sep 11 16:20:00.309522 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:20:00.309507    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60\": container with ID starting with a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60 not found: ID does not exist" containerID="a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60"
Sep 11 16:20:00.309554 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.309526    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60"} err="failed to get container status \"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60\": rpc error: code = NotFound desc = could not find container \"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60\": container with ID starting with a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60 not found: ID does not exist"
Sep 11 16:20:00.309554 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.309538    2561 scope.go:117] "RemoveContainer" containerID="73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c"
Sep 11 16:20:00.309730 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:20:00.309710    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c\": container with ID starting with 73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c not found: ID does not exist" containerID="73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c"
Sep 11 16:20:00.309761 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.309733    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c"} err="failed to get container status \"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c\": rpc error: code = NotFound desc = could not find container \"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c\": container with ID starting with 73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c not found: ID does not exist"
Sep 11 16:20:00.309761 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.309746    2561 scope.go:117] "RemoveContainer" containerID="dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5"
Sep 11 16:20:00.309970 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:20:00.309953    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5\": container with ID starting with dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5 not found: ID does not exist" containerID="dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5"
Sep 11 16:20:00.310008 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.309975    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5"} err="failed to get container status \"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5\": rpc error: code = NotFound desc = could not find container \"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5\": container with ID starting with dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5 not found: ID does not exist"
Sep 11 16:20:00.310008 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.309998    2561 scope.go:117] "RemoveContainer" containerID="91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde"
Sep 11 16:20:00.310289 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:20:00.310263    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde\": container with ID starting with 91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde not found: ID does not exist" containerID="91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde"
Sep 11 16:20:00.310382 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.310294    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde"} err="failed to get container status \"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde\": rpc error: code = NotFound desc = could not find container \"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde\": container with ID starting with 91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde not found: ID does not exist"
Sep 11 16:20:00.310382 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.310313    2561 scope.go:117] "RemoveContainer" containerID="04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b"
Sep 11 16:20:00.310566 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:20:00.310548    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b\": container with ID starting with 04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b not found: ID does not exist" containerID="04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b"
Sep 11 16:20:00.310606 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.310571    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b"} err="failed to get container status \"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b\": rpc error: code = NotFound desc = could not find container \"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b\": container with ID starting with 04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b not found: ID does not exist"
Sep 11 16:20:00.310606 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.310585    2561 scope.go:117] "RemoveContainer" containerID="62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4"
Sep 11 16:20:00.310860 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.310835    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4"} err="failed to get container status \"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4\": rpc error: code = NotFound desc = could not find container \"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4\": container with ID starting with 62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4 not found: ID does not exist"
Sep 11 16:20:00.310949 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.310860    2561 scope.go:117] "RemoveContainer" containerID="0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553"
Sep 11 16:20:00.311189 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.311159    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553"} err="failed to get container status \"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553\": rpc error: code = NotFound desc = could not find container \"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553\": container with ID starting with 0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553 not found: ID does not exist"
Sep 11 16:20:00.311189 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.311194    2561 scope.go:117] "RemoveContainer" containerID="a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60"
Sep 11 16:20:00.311531 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.311502    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60"} err="failed to get container status \"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60\": rpc error: code = NotFound desc = could not find container \"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60\": container with ID starting with a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60 not found: ID does not exist"
Sep 11 16:20:00.311596 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.311534    2561 scope.go:117] "RemoveContainer" containerID="73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c"
Sep 11 16:20:00.312719 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.312676    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c"} err="failed to get container status \"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c\": rpc error: code = NotFound desc = could not find container \"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c\": container with ID starting with 73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c not found: ID does not exist"
Sep 11 16:20:00.312719 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.312720    2561 scope.go:117] "RemoveContainer" containerID="dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5"
Sep 11 16:20:00.313042 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.313017    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5"} err="failed to get container status \"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5\": rpc error: code = NotFound desc = could not find container \"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5\": container with ID starting with dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5 not found: ID does not exist"
Sep 11 16:20:00.313108 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.313044    2561 scope.go:117] "RemoveContainer" containerID="91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde"
Sep 11 16:20:00.313380 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.313357    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde"} err="failed to get container status \"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde\": rpc error: code = NotFound desc = could not find container \"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde\": container with ID starting with 91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde not found: ID does not exist"
Sep 11 16:20:00.313463 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.313381    2561 scope.go:117] "RemoveContainer" containerID="04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b"
Sep 11 16:20:00.313654 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.313635    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b"} err="failed to get container status \"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b\": rpc error: code = NotFound desc = could not find container \"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b\": container with ID starting with 04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b not found: ID does not exist"
Sep 11 16:20:00.313708 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.313655    2561 scope.go:117] "RemoveContainer" containerID="62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4"
Sep 11 16:20:00.313939 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.313910    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4"} err="failed to get container status \"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4\": rpc error: code = NotFound desc = could not find container \"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4\": container with ID starting with 62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4 not found: ID does not exist"
Sep 11 16:20:00.313939 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.313931    2561 scope.go:117] "RemoveContainer" containerID="0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553"
Sep 11 16:20:00.314205 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.314161    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553"} err="failed to get container status \"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553\": rpc error: code = NotFound desc = could not find container \"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553\": container with ID starting with 0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553 not found: ID does not exist"
Sep 11 16:20:00.314252 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.314207    2561 scope.go:117] "RemoveContainer" containerID="a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60"
Sep 11 16:20:00.314367 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.314352    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:20:00.314444 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.314428    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60"} err="failed to get container status \"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60\": rpc error: code = NotFound desc = could not find container \"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60\": container with ID starting with a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60 not found: ID does not exist"
Sep 11 16:20:00.314486 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.314445    2561 scope.go:117] "RemoveContainer" containerID="73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c"
Sep 11 16:20:00.314659 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.314641    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c"} err="failed to get container status \"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c\": rpc error: code = NotFound desc = could not find container \"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c\": container with ID starting with 73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c not found: ID does not exist"
Sep 11 16:20:00.314712 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.314659    2561 scope.go:117] "RemoveContainer" containerID="dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5"
Sep 11 16:20:00.314893 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.314873    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5"} err="failed to get container status \"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5\": rpc error: code = NotFound desc = could not find container \"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5\": container with ID starting with dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5 not found: ID does not exist"
Sep 11 16:20:00.314944 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.314894    2561 scope.go:117] "RemoveContainer" containerID="91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde"
Sep 11 16:20:00.315091 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.315076    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde"} err="failed to get container status \"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde\": rpc error: code = NotFound desc = could not find container \"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde\": container with ID starting with 91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde not found: ID does not exist"
Sep 11 16:20:00.315138 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.315090    2561 scope.go:117] "RemoveContainer" containerID="04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b"
Sep 11 16:20:00.315327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.315301    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b"} err="failed to get container status \"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b\": rpc error: code = NotFound desc = could not find container \"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b\": container with ID starting with 04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b not found: ID does not exist"
Sep 11 16:20:00.315327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.315319    2561 scope.go:117] "RemoveContainer" containerID="62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4"
Sep 11 16:20:00.315545 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.315525    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4"} err="failed to get container status \"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4\": rpc error: code = NotFound desc = could not find container \"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4\": container with ID starting with 62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4 not found: ID does not exist"
Sep 11 16:20:00.315580 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.315546    2561 scope.go:117] "RemoveContainer" containerID="0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553"
Sep 11 16:20:00.315773 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.315755    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553"} err="failed to get container status \"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553\": rpc error: code = NotFound desc = could not find container \"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553\": container with ID starting with 0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553 not found: ID does not exist"
Sep 11 16:20:00.315773 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.315772    2561 scope.go:117] "RemoveContainer" containerID="a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60"
Sep 11 16:20:00.315971 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.315952    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60"} err="failed to get container status \"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60\": rpc error: code = NotFound desc = could not find container \"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60\": container with ID starting with a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60 not found: ID does not exist"
Sep 11 16:20:00.316012 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.315971    2561 scope.go:117] "RemoveContainer" containerID="73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c"
Sep 11 16:20:00.316166 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.316152    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c"} err="failed to get container status \"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c\": rpc error: code = NotFound desc = could not find container \"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c\": container with ID starting with 73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c not found: ID does not exist"
Sep 11 16:20:00.316166 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.316166    2561 scope.go:117] "RemoveContainer" containerID="dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5"
Sep 11 16:20:00.316425 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.316408    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5"} err="failed to get container status \"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5\": rpc error: code = NotFound desc = could not find container \"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5\": container with ID starting with dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5 not found: ID does not exist"
Sep 11 16:20:00.316425 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.316423    2561 scope.go:117] "RemoveContainer" containerID="91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde"
Sep 11 16:20:00.316606 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.316591    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde"} err="failed to get container status \"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde\": rpc error: code = NotFound desc = could not find container \"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde\": container with ID starting with 91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde not found: ID does not exist"
Sep 11 16:20:00.316606 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.316605    2561 scope.go:117] "RemoveContainer" containerID="04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b"
Sep 11 16:20:00.316829 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.316810    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b"} err="failed to get container status \"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b\": rpc error: code = NotFound desc = could not find container \"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b\": container with ID starting with 04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b not found: ID does not exist"
Sep 11 16:20:00.316884 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.316837    2561 scope.go:117] "RemoveContainer" containerID="62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4"
Sep 11 16:20:00.317043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.317029    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4"} err="failed to get container status \"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4\": rpc error: code = NotFound desc = could not find container \"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4\": container with ID starting with 62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4 not found: ID does not exist"
Sep 11 16:20:00.317043 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.317042    2561 scope.go:117] "RemoveContainer" containerID="0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553"
Sep 11 16:20:00.317261 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.317240    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553"} err="failed to get container status \"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553\": rpc error: code = NotFound desc = could not find container \"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553\": container with ID starting with 0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553 not found: ID does not exist"
Sep 11 16:20:00.317261 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.317258    2561 scope.go:117] "RemoveContainer" containerID="a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60"
Sep 11 16:20:00.317475 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.317456    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60"} err="failed to get container status \"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60\": rpc error: code = NotFound desc = could not find container \"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60\": container with ID starting with a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60 not found: ID does not exist"
Sep 11 16:20:00.317552 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.317477    2561 scope.go:117] "RemoveContainer" containerID="73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c"
Sep 11 16:20:00.317729 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.317713    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c"} err="failed to get container status \"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c\": rpc error: code = NotFound desc = could not find container \"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c\": container with ID starting with 73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c not found: ID does not exist"
Sep 11 16:20:00.317774 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.317732    2561 scope.go:117] "RemoveContainer" containerID="dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5"
Sep 11 16:20:00.317935 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.317918    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5"} err="failed to get container status \"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5\": rpc error: code = NotFound desc = could not find container \"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5\": container with ID starting with dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5 not found: ID does not exist"
Sep 11 16:20:00.317994 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.317936    2561 scope.go:117] "RemoveContainer" containerID="91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde"
Sep 11 16:20:00.318160 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.318144    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde"} err="failed to get container status \"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde\": rpc error: code = NotFound desc = could not find container \"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde\": container with ID starting with 91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde not found: ID does not exist"
Sep 11 16:20:00.318236 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.318160    2561 scope.go:117] "RemoveContainer" containerID="04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b"
Sep 11 16:20:00.318446 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.318428    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b"} err="failed to get container status \"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b\": rpc error: code = NotFound desc = could not find container \"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b\": container with ID starting with 04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b not found: ID does not exist"
Sep 11 16:20:00.318493 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.318447    2561 scope.go:117] "RemoveContainer" containerID="62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4"
Sep 11 16:20:00.318687 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.318671    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4"} err="failed to get container status \"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4\": rpc error: code = NotFound desc = could not find container \"62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4\": container with ID starting with 62b1bf6c03b3da772f29b786cb9d68b4090d37df903dedfdb2a5e2f39ca3d0a4 not found: ID does not exist"
Sep 11 16:20:00.318755 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.318690    2561 scope.go:117] "RemoveContainer" containerID="0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553"
Sep 11 16:20:00.318892 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.318877    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553"} err="failed to get container status \"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553\": rpc error: code = NotFound desc = could not find container \"0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553\": container with ID starting with 0e377b32ebb6a7ddb83519fc064b551e51aa19f393da8c1e41d67b46937c3553 not found: ID does not exist"
Sep 11 16:20:00.318934 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.318892    2561 scope.go:117] "RemoveContainer" containerID="a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60"
Sep 11 16:20:00.319108 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.319091    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60"} err="failed to get container status \"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60\": rpc error: code = NotFound desc = could not find container \"a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60\": container with ID starting with a66450494dc1ddeafbe6adf0cbe6b6ea1334ca7d82227c546b61c50f8733ed60 not found: ID does not exist"
Sep 11 16:20:00.319187 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.319110    2561 scope.go:117] "RemoveContainer" containerID="73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c"
Sep 11 16:20:00.319345 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.319328    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c"} err="failed to get container status \"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c\": rpc error: code = NotFound desc = could not find container \"73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c\": container with ID starting with 73f3ec41972d8232cdb0687bf32526d4ae84e910f8e1c743b4e5c5f192bfc56c not found: ID does not exist"
Sep 11 16:20:00.319395 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.319345    2561 scope.go:117] "RemoveContainer" containerID="dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5"
Sep 11 16:20:00.319589 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.319572    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5"} err="failed to get container status \"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5\": rpc error: code = NotFound desc = could not find container \"dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5\": container with ID starting with dda659c073c5f6f94e3b5cfc3656c209b65c1ab12beeb210631327845c3a1ae5 not found: ID does not exist"
Sep 11 16:20:00.319664 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.319590    2561 scope.go:117] "RemoveContainer" containerID="91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde"
Sep 11 16:20:00.319791 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.319775    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde"} err="failed to get container status \"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde\": rpc error: code = NotFound desc = could not find container \"91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde\": container with ID starting with 91d77f813c566780f9da3df90f97dee1b18211be6f89cebcd3d96e02ff9b5fde not found: ID does not exist"
Sep 11 16:20:00.319849 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.319792    2561 scope.go:117] "RemoveContainer" containerID="04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b"
Sep 11 16:20:00.320022 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.320003    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b"} err="failed to get container status \"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b\": rpc error: code = NotFound desc = could not find container \"04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b\": container with ID starting with 04813cc8f986ea84ba01b1b0c72dc9893f8c00f7d2c276cadffc32eff54e017b not found: ID does not exist"
Sep 11 16:20:00.347307 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347237    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:20:00.347576 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347564    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="kube-rbac-proxy"
Sep 11 16:20:00.347622 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347578    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="kube-rbac-proxy"
Sep 11 16:20:00.347622 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347589    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="config-reloader"
Sep 11 16:20:00.347622 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347596    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="config-reloader"
Sep 11 16:20:00.347622 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347604    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="thanos-sidecar"
Sep 11 16:20:00.347622 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347609    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="thanos-sidecar"
Sep 11 16:20:00.347622 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347616    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="kube-rbac-proxy-web"
Sep 11 16:20:00.347622 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347620    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="kube-rbac-proxy-web"
Sep 11 16:20:00.347831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347631    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="prometheus"
Sep 11 16:20:00.347831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347636    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="prometheus"
Sep 11 16:20:00.347831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347647    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="init-config-reloader"
Sep 11 16:20:00.347831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347652    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="init-config-reloader"
Sep 11 16:20:00.347831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347659    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="kube-rbac-proxy-thanos"
Sep 11 16:20:00.347831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347664    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="kube-rbac-proxy-thanos"
Sep 11 16:20:00.347831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347711    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="kube-rbac-proxy-web"
Sep 11 16:20:00.347831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347718    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="prometheus"
Sep 11 16:20:00.347831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347725    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="kube-rbac-proxy-thanos"
Sep 11 16:20:00.347831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347732    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="config-reloader"
Sep 11 16:20:00.347831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347739    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="thanos-sidecar"
Sep 11 16:20:00.347831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.347745    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" containerName="kube-rbac-proxy"
Sep 11 16:20:00.353046 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.353023    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.355572 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.355553    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7ds8r\""
Sep 11 16:20:00.356555 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.356537    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 16:20:00.356839 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.356825    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 16:20:00.357446 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.357430    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 16:20:00.357702 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.357685    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 16:20:00.358257 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.358242    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 16:20:00.358327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.358272    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 16:20:00.358327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.358277    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 16:20:00.358517 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.358501    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 16:20:00.358571 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.358562    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4gqm6e1v16sc5\""
Sep 11 16:20:00.358649 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.358633    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 16:20:00.359395 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.359376    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 16:20:00.359585 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.359403    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 16:20:00.359865 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.359843    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 16:20:00.359968 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.359906    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 16:20:00.365309 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.365288    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 16:20:00.367324 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.367304    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 16:20:00.386662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.386637    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:20:00.512433 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.512433 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512438    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.512700 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512464    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.512700 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512551    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/786b414e-7f35-4733-a87c-f0b7c32928b9-config-out\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.512700 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512606    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.512700 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512638    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.512700 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512662    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.512700 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.512901 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512730    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/786b414e-7f35-4733-a87c-f0b7c32928b9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.512901 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512757    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.512901 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-web-config\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.512901 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512811    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/786b414e-7f35-4733-a87c-f0b7c32928b9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.512901 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512830    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-config\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.512901 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512877    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.513162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512905    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jl9b\" (UniqueName: \"kubernetes.io/projected/786b414e-7f35-4733-a87c-f0b7c32928b9-kube-api-access-5jl9b\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.513162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512938    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.513162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.512990    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.513162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.513022    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.513162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.513048    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.513162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.513092    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.613949 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.613853    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jl9b\" (UniqueName: \"kubernetes.io/projected/786b414e-7f35-4733-a87c-f0b7c32928b9-kube-api-access-5jl9b\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.613949 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.613895    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.613949 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.613928    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614243 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.613955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614243 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.613982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614243 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614005    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614243 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614027    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614243 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614056    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614243 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614092    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614243 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614121    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/786b414e-7f35-4733-a87c-f0b7c32928b9-config-out\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614243 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614166    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614614 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614614 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614304    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614614 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614336    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614614 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/786b414e-7f35-4733-a87c-f0b7c32928b9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614614 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614402    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614614 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614437    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-web-config\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614614 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/786b414e-7f35-4733-a87c-f0b7c32928b9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614614 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614499    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-config\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.614614 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614541    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.615023 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614735    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.615023 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614782    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.615023 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.614822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.615310 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.615287    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.615405 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.615315    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.615614 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.615588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/786b414e-7f35-4733-a87c-f0b7c32928b9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.617522 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.617220    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.617615 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.617552    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.617675 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.617605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.617896 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.617867    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.617980 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.617957    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/786b414e-7f35-4733-a87c-f0b7c32928b9-config-out\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.618281 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.618247    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.618419 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.618395    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.618515 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.618473    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/786b414e-7f35-4733-a87c-f0b7c32928b9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.618992 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.618970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.620082 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.620061    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/786b414e-7f35-4733-a87c-f0b7c32928b9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.620273 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.620257    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-web-config\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.620349 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.620321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-config\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.620404 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.620345    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/786b414e-7f35-4733-a87c-f0b7c32928b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.623571 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.623546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jl9b\" (UniqueName: \"kubernetes.io/projected/786b414e-7f35-4733-a87c-f0b7c32928b9-kube-api-access-5jl9b\") pod \"prometheus-k8s-0\" (UID: \"786b414e-7f35-4733-a87c-f0b7c32928b9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.663533 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.663493    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:20:00.952034 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:00.952007    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 16:20:00.954218 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:20:00.954194    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786b414e_7f35_4733_a87c_f0b7c32928b9.slice/crio-c305f92a61e53617db6613f2105010efba3ea8a55eb52391f97d5d907a059f4c WatchSource:0}: Error finding container c305f92a61e53617db6613f2105010efba3ea8a55eb52391f97d5d907a059f4c: Status 404 returned error can't find the container with id c305f92a61e53617db6613f2105010efba3ea8a55eb52391f97d5d907a059f4c
Sep 11 16:20:01.260430 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:01.260335    2561 generic.go:356] "Generic (PLEG): container finished" podID="786b414e-7f35-4733-a87c-f0b7c32928b9" containerID="a229fa813fe09509403440a6fd4d5e32a35eb9bac0a9455286a3d8f519e5b51b" exitCode=0
Sep 11 16:20:01.260785 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:01.260424    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"786b414e-7f35-4733-a87c-f0b7c32928b9","Type":"ContainerDied","Data":"a229fa813fe09509403440a6fd4d5e32a35eb9bac0a9455286a3d8f519e5b51b"}
Sep 11 16:20:01.260785 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:01.260493    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"786b414e-7f35-4733-a87c-f0b7c32928b9","Type":"ContainerStarted","Data":"c305f92a61e53617db6613f2105010efba3ea8a55eb52391f97d5d907a059f4c"}
Sep 11 16:20:01.322455 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:01.322418    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d12d629-e6a5-4868-8554-ff0a8318f6e6" path="/var/lib/kubelet/pods/3d12d629-e6a5-4868-8554-ff0a8318f6e6/volumes"
Sep 11 16:20:02.269379 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:02.269345    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"786b414e-7f35-4733-a87c-f0b7c32928b9","Type":"ContainerStarted","Data":"964f19350c3a2976db3f068612dad3ac35a6cd312c434612839603c695ef76c8"}
Sep 11 16:20:02.269790 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:02.269466    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"786b414e-7f35-4733-a87c-f0b7c32928b9","Type":"ContainerStarted","Data":"8a22675ced25a94296e8a89264453369c76304fffe9cd25238165d5970f3052f"}
Sep 11 16:20:02.269790 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:02.269516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"786b414e-7f35-4733-a87c-f0b7c32928b9","Type":"ContainerStarted","Data":"b621594d1f0260e24a34a70e0cfe15444473e3abedbae8811ee8833ab7214448"}
Sep 11 16:20:02.269790 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:02.269555    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"786b414e-7f35-4733-a87c-f0b7c32928b9","Type":"ContainerStarted","Data":"b1d1b0106cdfb98134366c0c1ff8b41d5249ce98d6904188600d30dfa5328633"}
Sep 11 16:20:02.269790 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:02.269595    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"786b414e-7f35-4733-a87c-f0b7c32928b9","Type":"ContainerStarted","Data":"53c113926978c4e67610fa91b5aa595ff6efcbd805367d344de07158791acf5d"}
Sep 11 16:20:02.269790 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:02.269633    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"786b414e-7f35-4733-a87c-f0b7c32928b9","Type":"ContainerStarted","Data":"c7c402b36e972e240d505cb013fb4a48a272f4c42d9ec85502e00921935cbc51"}
Sep 11 16:20:02.310522 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:02.310469    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.2898247339999998 podStartE2EDuration="2.31045169s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.626956ms" isStatefulPod=false podCreationTimestamp="2026-09-11 16:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:20:02.307812832 +0000 UTC m=+207.647906889" watchObservedRunningTime="2026-09-11 16:20:02.31045169 +0000 UTC m=+207.650545758"
Sep 11 16:20:05.664054 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:20:05.663988    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:21:00.663790 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:21:00.663683    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:21:00.679340 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:21:00.679316    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:21:01.537984 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:21:01.537957    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 16:21:35.101084 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:21:35.101059    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:21:35.103837 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:21:35.103814    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 16:21:35.240754 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:21:35.240728    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:25:22.305923 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.305889    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-j8rk2"]
Sep 11 16:25:22.309416 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.309400    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-j8rk2"
Sep 11 16:25:22.311589 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.311568    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 11 16:25:22.311696 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.311628    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 11 16:25:22.312220 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.312201    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q9d2b\""
Sep 11 16:25:22.312301 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.312217    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 11 16:25:22.312301 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.312217    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 11 16:25:22.316931 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.316908    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-j8rk2"]
Sep 11 16:25:22.362421 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.362389    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1a90201-ed37-43bd-a91f-27460661fa37-model-cache\") pod \"s3-init-j8rk2\" (UID: \"f1a90201-ed37-43bd-a91f-27460661fa37\") " pod="kserve/s3-init-j8rk2"
Sep 11 16:25:22.362591 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.362476    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25psj\" (UniqueName: \"kubernetes.io/projected/f1a90201-ed37-43bd-a91f-27460661fa37-kube-api-access-25psj\") pod \"s3-init-j8rk2\" (UID: \"f1a90201-ed37-43bd-a91f-27460661fa37\") " pod="kserve/s3-init-j8rk2"
Sep 11 16:25:22.463637 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.463601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1a90201-ed37-43bd-a91f-27460661fa37-model-cache\") pod \"s3-init-j8rk2\" (UID: \"f1a90201-ed37-43bd-a91f-27460661fa37\") " pod="kserve/s3-init-j8rk2"
Sep 11 16:25:22.463780 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.463756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25psj\" (UniqueName: \"kubernetes.io/projected/f1a90201-ed37-43bd-a91f-27460661fa37-kube-api-access-25psj\") pod \"s3-init-j8rk2\" (UID: \"f1a90201-ed37-43bd-a91f-27460661fa37\") " pod="kserve/s3-init-j8rk2"
Sep 11 16:25:22.463932 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.463916    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1a90201-ed37-43bd-a91f-27460661fa37-model-cache\") pod \"s3-init-j8rk2\" (UID: \"f1a90201-ed37-43bd-a91f-27460661fa37\") " pod="kserve/s3-init-j8rk2"
Sep 11 16:25:22.473926 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.473904    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25psj\" (UniqueName: \"kubernetes.io/projected/f1a90201-ed37-43bd-a91f-27460661fa37-kube-api-access-25psj\") pod \"s3-init-j8rk2\" (UID: \"f1a90201-ed37-43bd-a91f-27460661fa37\") " pod="kserve/s3-init-j8rk2"
Sep 11 16:25:22.629273 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.629193    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-j8rk2"
Sep 11 16:25:22.899025 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.898974    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-j8rk2"]
Sep 11 16:25:22.901849 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:25:22.901820    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1a90201_ed37_43bd_a91f_27460661fa37.slice/crio-8d354cc4f56ef37365978445f74c6748bebe39e188d1cc1fcccf01cdfff76258 WatchSource:0}: Error finding container 8d354cc4f56ef37365978445f74c6748bebe39e188d1cc1fcccf01cdfff76258: Status 404 returned error can't find the container with id 8d354cc4f56ef37365978445f74c6748bebe39e188d1cc1fcccf01cdfff76258
Sep 11 16:25:22.903683 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:22.903665    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:25:23.637928 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:23.637892    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j8rk2" event={"ID":"f1a90201-ed37-43bd-a91f-27460661fa37","Type":"ContainerStarted","Data":"8d354cc4f56ef37365978445f74c6748bebe39e188d1cc1fcccf01cdfff76258"}
Sep 11 16:25:28.668981 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:28.668873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j8rk2" event={"ID":"f1a90201-ed37-43bd-a91f-27460661fa37","Type":"ContainerStarted","Data":"506c61b1e40d53d9c894366b9f94c111a5fac7df7e7671d10474e120852cd622"}
Sep 11 16:25:30.678527 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:30.678490    2561 generic.go:356] "Generic (PLEG): container finished" podID="f1a90201-ed37-43bd-a91f-27460661fa37" containerID="506c61b1e40d53d9c894366b9f94c111a5fac7df7e7671d10474e120852cd622" exitCode=0
Sep 11 16:25:30.678888 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:30.678530    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j8rk2" event={"ID":"f1a90201-ed37-43bd-a91f-27460661fa37","Type":"ContainerDied","Data":"506c61b1e40d53d9c894366b9f94c111a5fac7df7e7671d10474e120852cd622"}
Sep 11 16:25:35.130118 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:35.130084    2561 scope.go:117] "RemoveContainer" containerID="9eda93326c78d4c065565a192c958e76b1ac70d9742659f3b07a5ad7f4ac75f2"
Sep 11 16:25:35.138111 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:35.138085    2561 scope.go:117] "RemoveContainer" containerID="509be74cc1b0b2dd9f97b180012b369a7573fba5a722dc95aa1eca1d0d1a3d41"
Sep 11 16:25:35.145553 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:35.145534    2561 scope.go:117] "RemoveContainer" containerID="fe8084d671abe41c32081f7e2e1f0f5fffa3374c301175af9bb07751746321de"
Sep 11 16:25:35.152832 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:35.152814    2561 scope.go:117] "RemoveContainer" containerID="6817fd748c356811980e7a0bd4bb70c8428f26102d7c315f1c6dc47f09a0b82e"
Sep 11 16:25:35.159919 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:35.159899    2561 scope.go:117] "RemoveContainer" containerID="b722800fad55a579c288129e596d4676d75df01d4fa2e87e1836c46ee326c444"
Sep 11 16:25:35.166775 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:35.166756    2561 scope.go:117] "RemoveContainer" containerID="169b45da9fd2b2022bb5f31f61e2e94ef00152afae489a38c31f36bf6f57d13d"
Sep 11 16:25:35.174728 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:35.174713    2561 scope.go:117] "RemoveContainer" containerID="5e996b81eff51bbd8ee92b70613bd0fa19ba4fd016f9209591cee2efc8a01b23"
Sep 11 16:25:35.707284 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:35.707250    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j8rk2" event={"ID":"f1a90201-ed37-43bd-a91f-27460661fa37","Type":"ContainerStarted","Data":"3c7ec4d1e3095146a6da54aea6e2e32d485d2af0e39850bcdabdc87af9ad3558"}
Sep 11 16:25:35.726941 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:25:35.726891    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-j8rk2" podStartSLOduration=2.4662774499999998 podStartE2EDuration="13.726874136s" totalImagesPullingTime="9.520262145s" totalInitContainerRuntime="1.740334541s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:25:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:25:35.725147076 +0000 UTC m=+541.065241139" watchObservedRunningTime="2026-09-11 16:25:35.726874136 +0000 UTC m=+541.066968180"
Sep 11 16:26:35.131272 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:26:35.131242    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:26:35.268573 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:26:35.268534    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:26:39.987728 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:26:39.987693    2561 generic.go:356] "Generic (PLEG): container finished" podID="f1a90201-ed37-43bd-a91f-27460661fa37" containerID="3c7ec4d1e3095146a6da54aea6e2e32d485d2af0e39850bcdabdc87af9ad3558" exitCode=0
Sep 11 16:26:39.988152 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:26:39.987737    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j8rk2" event={"ID":"f1a90201-ed37-43bd-a91f-27460661fa37","Type":"ContainerDied","Data":"3c7ec4d1e3095146a6da54aea6e2e32d485d2af0e39850bcdabdc87af9ad3558"}
Sep 11 16:26:41.111423 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:26:41.111398    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-j8rk2"
Sep 11 16:26:41.239588 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:26:41.239552    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25psj\" (UniqueName: \"kubernetes.io/projected/f1a90201-ed37-43bd-a91f-27460661fa37-kube-api-access-25psj\") pod \"f1a90201-ed37-43bd-a91f-27460661fa37\" (UID: \"f1a90201-ed37-43bd-a91f-27460661fa37\") "
Sep 11 16:26:41.239781 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:26:41.239603    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1a90201-ed37-43bd-a91f-27460661fa37-model-cache\") pod \"f1a90201-ed37-43bd-a91f-27460661fa37\" (UID: \"f1a90201-ed37-43bd-a91f-27460661fa37\") "
Sep 11 16:26:41.241892 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:26:41.241863    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a90201-ed37-43bd-a91f-27460661fa37-kube-api-access-25psj" (OuterVolumeSpecName: "kube-api-access-25psj") pod "f1a90201-ed37-43bd-a91f-27460661fa37" (UID: "f1a90201-ed37-43bd-a91f-27460661fa37"). InnerVolumeSpecName "kube-api-access-25psj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:26:41.257190 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:26:41.257141    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a90201-ed37-43bd-a91f-27460661fa37-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1a90201-ed37-43bd-a91f-27460661fa37" (UID: "f1a90201-ed37-43bd-a91f-27460661fa37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:26:41.340624 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:26:41.340588    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25psj\" (UniqueName: \"kubernetes.io/projected/f1a90201-ed37-43bd-a91f-27460661fa37-kube-api-access-25psj\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:26:41.340624 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:26:41.340616    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1a90201-ed37-43bd-a91f-27460661fa37-model-cache\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:26:41.997520 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:26:41.997495    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-j8rk2"
Sep 11 16:26:41.997687 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:26:41.997488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-j8rk2" event={"ID":"f1a90201-ed37-43bd-a91f-27460661fa37","Type":"ContainerDied","Data":"8d354cc4f56ef37365978445f74c6748bebe39e188d1cc1fcccf01cdfff76258"}
Sep 11 16:26:41.997687 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:26:41.997610    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d354cc4f56ef37365978445f74c6748bebe39e188d1cc1fcccf01cdfff76258"
Sep 11 16:27:19.233512 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.233475    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-9ztlx"]
Sep 11 16:27:19.233927 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.233812    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1a90201-ed37-43bd-a91f-27460661fa37" containerName="s3-init"
Sep 11 16:27:19.233927 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.233824    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1a90201-ed37-43bd-a91f-27460661fa37" containerName="s3-init"
Sep 11 16:27:19.233927 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.233833    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1a90201-ed37-43bd-a91f-27460661fa37" containerName="download-hf-model"
Sep 11 16:27:19.233927 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.233838    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1a90201-ed37-43bd-a91f-27460661fa37" containerName="download-hf-model"
Sep 11 16:27:19.233927 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.233898    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1a90201-ed37-43bd-a91f-27460661fa37" containerName="s3-init"
Sep 11 16:27:19.236991 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.236976    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-9ztlx"
Sep 11 16:27:19.239334 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.239311    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 11 16:27:19.239463 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.239355    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 11 16:27:19.239846 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.239830    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q9d2b\""
Sep 11 16:27:19.239929 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.239867    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 11 16:27:19.239929 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.239895    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 11 16:27:19.245696 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.245674    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-9ztlx"]
Sep 11 16:27:19.377927 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.377889    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce8738b9-903b-4009-9bf3-49de8955d907-model-cache\") pod \"s3-tls-init-custom-9ztlx\" (UID: \"ce8738b9-903b-4009-9bf3-49de8955d907\") " pod="kserve/s3-tls-init-custom-9ztlx"
Sep 11 16:27:19.377927 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.377933    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttgpn\" (UniqueName: \"kubernetes.io/projected/ce8738b9-903b-4009-9bf3-49de8955d907-kube-api-access-ttgpn\") pod \"s3-tls-init-custom-9ztlx\" (UID: \"ce8738b9-903b-4009-9bf3-49de8955d907\") " pod="kserve/s3-tls-init-custom-9ztlx"
Sep 11 16:27:19.479047 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.479012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce8738b9-903b-4009-9bf3-49de8955d907-model-cache\") pod \"s3-tls-init-custom-9ztlx\" (UID: \"ce8738b9-903b-4009-9bf3-49de8955d907\") " pod="kserve/s3-tls-init-custom-9ztlx"
Sep 11 16:27:19.479047 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.479047    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgpn\" (UniqueName: \"kubernetes.io/projected/ce8738b9-903b-4009-9bf3-49de8955d907-kube-api-access-ttgpn\") pod \"s3-tls-init-custom-9ztlx\" (UID: \"ce8738b9-903b-4009-9bf3-49de8955d907\") " pod="kserve/s3-tls-init-custom-9ztlx"
Sep 11 16:27:19.479446 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.479426    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce8738b9-903b-4009-9bf3-49de8955d907-model-cache\") pod \"s3-tls-init-custom-9ztlx\" (UID: \"ce8738b9-903b-4009-9bf3-49de8955d907\") " pod="kserve/s3-tls-init-custom-9ztlx"
Sep 11 16:27:19.493306 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.493242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgpn\" (UniqueName: \"kubernetes.io/projected/ce8738b9-903b-4009-9bf3-49de8955d907-kube-api-access-ttgpn\") pod \"s3-tls-init-custom-9ztlx\" (UID: \"ce8738b9-903b-4009-9bf3-49de8955d907\") " pod="kserve/s3-tls-init-custom-9ztlx"
Sep 11 16:27:19.553417 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.553387    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-9ztlx"
Sep 11 16:27:19.826794 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:19.826655    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-9ztlx"]
Sep 11 16:27:19.831936 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:27:19.831908    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce8738b9_903b_4009_9bf3_49de8955d907.slice/crio-7d4e5d57c88f329c4a5598162ce5f5f26f22a556b64e497a9180a1ecedd0ebeb WatchSource:0}: Error finding container 7d4e5d57c88f329c4a5598162ce5f5f26f22a556b64e497a9180a1ecedd0ebeb: Status 404 returned error can't find the container with id 7d4e5d57c88f329c4a5598162ce5f5f26f22a556b64e497a9180a1ecedd0ebeb
Sep 11 16:27:20.159258 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:20.159217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-9ztlx" event={"ID":"ce8738b9-903b-4009-9bf3-49de8955d907","Type":"ContainerStarted","Data":"cec0c0951ad5bdbebbee98d715ba619a7ec70feea611f4686f9a670ed811d385"}
Sep 11 16:27:20.159450 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:20.159278    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-9ztlx" event={"ID":"ce8738b9-903b-4009-9bf3-49de8955d907","Type":"ContainerStarted","Data":"7d4e5d57c88f329c4a5598162ce5f5f26f22a556b64e497a9180a1ecedd0ebeb"}
Sep 11 16:27:22.170013 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:22.169981    2561 generic.go:356] "Generic (PLEG): container finished" podID="ce8738b9-903b-4009-9bf3-49de8955d907" containerID="cec0c0951ad5bdbebbee98d715ba619a7ec70feea611f4686f9a670ed811d385" exitCode=0
Sep 11 16:27:22.170413 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:22.170062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-9ztlx" event={"ID":"ce8738b9-903b-4009-9bf3-49de8955d907","Type":"ContainerDied","Data":"cec0c0951ad5bdbebbee98d715ba619a7ec70feea611f4686f9a670ed811d385"}
Sep 11 16:27:23.176075 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:23.176042    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-9ztlx" event={"ID":"ce8738b9-903b-4009-9bf3-49de8955d907","Type":"ContainerStarted","Data":"8797462a727ac421cfb9ff8235d2243b83edc75570b1709a9255694385305266"}
Sep 11 16:27:23.195604 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:23.195546    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-9ztlx" podStartSLOduration=2.311585411 podStartE2EDuration="4.19552971s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.883944299s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:27:23.193949008 +0000 UTC m=+648.534043063" watchObservedRunningTime="2026-09-11 16:27:23.19552971 +0000 UTC m=+648.535623753"
Sep 11 16:27:29.208430 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:29.208395    2561 generic.go:356] "Generic (PLEG): container finished" podID="ce8738b9-903b-4009-9bf3-49de8955d907" containerID="8797462a727ac421cfb9ff8235d2243b83edc75570b1709a9255694385305266" exitCode=0
Sep 11 16:27:29.208807 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:29.208467    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-9ztlx" event={"ID":"ce8738b9-903b-4009-9bf3-49de8955d907","Type":"ContainerDied","Data":"8797462a727ac421cfb9ff8235d2243b83edc75570b1709a9255694385305266"}
Sep 11 16:27:30.356009 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:30.355985    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-9ztlx"
Sep 11 16:27:30.481644 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:30.481537    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttgpn\" (UniqueName: \"kubernetes.io/projected/ce8738b9-903b-4009-9bf3-49de8955d907-kube-api-access-ttgpn\") pod \"ce8738b9-903b-4009-9bf3-49de8955d907\" (UID: \"ce8738b9-903b-4009-9bf3-49de8955d907\") "
Sep 11 16:27:30.481644 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:30.481654    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce8738b9-903b-4009-9bf3-49de8955d907-model-cache\") pod \"ce8738b9-903b-4009-9bf3-49de8955d907\" (UID: \"ce8738b9-903b-4009-9bf3-49de8955d907\") "
Sep 11 16:27:30.483635 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:30.483589    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8738b9-903b-4009-9bf3-49de8955d907-kube-api-access-ttgpn" (OuterVolumeSpecName: "kube-api-access-ttgpn") pod "ce8738b9-903b-4009-9bf3-49de8955d907" (UID: "ce8738b9-903b-4009-9bf3-49de8955d907"). InnerVolumeSpecName "kube-api-access-ttgpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:30.508291 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:30.508255    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8738b9-903b-4009-9bf3-49de8955d907-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce8738b9-903b-4009-9bf3-49de8955d907" (UID: "ce8738b9-903b-4009-9bf3-49de8955d907"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:27:30.583167 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:30.583135    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce8738b9-903b-4009-9bf3-49de8955d907-model-cache\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:27:30.583167 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:30.583164    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttgpn\" (UniqueName: \"kubernetes.io/projected/ce8738b9-903b-4009-9bf3-49de8955d907-kube-api-access-ttgpn\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:27:31.219268 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:31.219240    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-9ztlx"
Sep 11 16:27:31.219268 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:31.219263    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-9ztlx" event={"ID":"ce8738b9-903b-4009-9bf3-49de8955d907","Type":"ContainerDied","Data":"7d4e5d57c88f329c4a5598162ce5f5f26f22a556b64e497a9180a1ecedd0ebeb"}
Sep 11 16:27:31.219465 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:31.219294    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d4e5d57c88f329c4a5598162ce5f5f26f22a556b64e497a9180a1ecedd0ebeb"
Sep 11 16:27:34.882709 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:34.882673    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-f2pml"]
Sep 11 16:27:34.883072 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:34.883009    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce8738b9-903b-4009-9bf3-49de8955d907" containerName="download-hf-model"
Sep 11 16:27:34.883072 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:34.883020    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce8738b9-903b-4009-9bf3-49de8955d907" containerName="download-hf-model"
Sep 11 16:27:34.883072 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:34.883028    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce8738b9-903b-4009-9bf3-49de8955d907" containerName="s3-tls-init-custom"
Sep 11 16:27:34.883072 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:34.883033    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce8738b9-903b-4009-9bf3-49de8955d907" containerName="s3-tls-init-custom"
Sep 11 16:27:34.883228 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:34.883094    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce8738b9-903b-4009-9bf3-49de8955d907" containerName="s3-tls-init-custom"
Sep 11 16:27:34.886418 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:34.886399    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-f2pml"
Sep 11 16:27:34.888898 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:34.888875    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q9d2b\""
Sep 11 16:27:34.889104 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:34.889089    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 11 16:27:34.889272 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:34.889254    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 11 16:27:34.889342 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:34.889283    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 11 16:27:34.889576 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:34.889357    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 11 16:27:34.897105 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:34.897080    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-f2pml"]
Sep 11 16:27:34.922037 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:34.922007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89vp2\" (UniqueName: \"kubernetes.io/projected/a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b-kube-api-access-89vp2\") pod \"s3-tls-init-serving-f2pml\" (UID: \"a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b\") " pod="kserve/s3-tls-init-serving-f2pml"
Sep 11 16:27:34.922207 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:34.922059    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b-model-cache\") pod \"s3-tls-init-serving-f2pml\" (UID: \"a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b\") " pod="kserve/s3-tls-init-serving-f2pml"
Sep 11 16:27:35.023087 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:35.023050    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89vp2\" (UniqueName: \"kubernetes.io/projected/a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b-kube-api-access-89vp2\") pod \"s3-tls-init-serving-f2pml\" (UID: \"a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b\") " pod="kserve/s3-tls-init-serving-f2pml"
Sep 11 16:27:35.023265 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:35.023106    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b-model-cache\") pod \"s3-tls-init-serving-f2pml\" (UID: \"a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b\") " pod="kserve/s3-tls-init-serving-f2pml"
Sep 11 16:27:35.023469 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:35.023452    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b-model-cache\") pod \"s3-tls-init-serving-f2pml\" (UID: \"a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b\") " pod="kserve/s3-tls-init-serving-f2pml"
Sep 11 16:27:35.032996 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:35.032973    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89vp2\" (UniqueName: \"kubernetes.io/projected/a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b-kube-api-access-89vp2\") pod \"s3-tls-init-serving-f2pml\" (UID: \"a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b\") " pod="kserve/s3-tls-init-serving-f2pml"
Sep 11 16:27:35.207144 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:35.207117    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q9d2b\""
Sep 11 16:27:35.215445 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:35.215427    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-f2pml"
Sep 11 16:27:35.487872 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:35.487791    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-f2pml"]
Sep 11 16:27:35.491544 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:27:35.491517    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0bf062c_70a6_4f68_8e4b_74aa7a5ce86b.slice/crio-e81ab43af7c9b2307b8ed59c23e0c890123b04feff9f2299d40106c37a7d0afd WatchSource:0}: Error finding container e81ab43af7c9b2307b8ed59c23e0c890123b04feff9f2299d40106c37a7d0afd: Status 404 returned error can't find the container with id e81ab43af7c9b2307b8ed59c23e0c890123b04feff9f2299d40106c37a7d0afd
Sep 11 16:27:35.495388 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:35.495368    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 11 16:27:36.243619 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:36.243582    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f2pml" event={"ID":"a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b","Type":"ContainerStarted","Data":"2e2effb96fd25329249d88cd27256b8061a15ffa85d8123f5378dfcb466c6695"}
Sep 11 16:27:36.243619 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:36.243625    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f2pml" event={"ID":"a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b","Type":"ContainerStarted","Data":"e81ab43af7c9b2307b8ed59c23e0c890123b04feff9f2299d40106c37a7d0afd"}
Sep 11 16:27:38.254030 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:38.253986    2561 generic.go:356] "Generic (PLEG): container finished" podID="a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b" containerID="2e2effb96fd25329249d88cd27256b8061a15ffa85d8123f5378dfcb466c6695" exitCode=0
Sep 11 16:27:38.254430 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:38.254060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f2pml" event={"ID":"a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b","Type":"ContainerDied","Data":"2e2effb96fd25329249d88cd27256b8061a15ffa85d8123f5378dfcb466c6695"}
Sep 11 16:27:38.263203 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:38.263184    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 11 16:27:39.260905 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:39.260871    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f2pml" event={"ID":"a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b","Type":"ContainerStarted","Data":"c8af10ab12e34c74fc54b8f5a8208c80a1a45fc5eb68d8527e46085ffbe11882"}
Sep 11 16:27:39.279473 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:39.279425    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-f2pml" podStartSLOduration=3.278795217 podStartE2EDuration="5.279409182s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.000613965s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:27:39.278943387 +0000 UTC m=+664.619037433" watchObservedRunningTime="2026-09-11 16:27:39.279409182 +0000 UTC m=+664.619503253"
Sep 11 16:27:44.285786 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:44.285754    2561 generic.go:356] "Generic (PLEG): container finished" podID="a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b" containerID="c8af10ab12e34c74fc54b8f5a8208c80a1a45fc5eb68d8527e46085ffbe11882" exitCode=0
Sep 11 16:27:44.286221 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:44.285831    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f2pml" event={"ID":"a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b","Type":"ContainerDied","Data":"c8af10ab12e34c74fc54b8f5a8208c80a1a45fc5eb68d8527e46085ffbe11882"}
Sep 11 16:27:45.428685 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:45.428657    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-f2pml"
Sep 11 16:27:45.501227 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:45.501202    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b-model-cache\") pod \"a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b\" (UID: \"a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b\") "
Sep 11 16:27:45.501391 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:45.501259    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89vp2\" (UniqueName: \"kubernetes.io/projected/a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b-kube-api-access-89vp2\") pod \"a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b\" (UID: \"a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b\") "
Sep 11 16:27:45.503256 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:45.503228    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b-kube-api-access-89vp2" (OuterVolumeSpecName: "kube-api-access-89vp2") pod "a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b" (UID: "a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b"). InnerVolumeSpecName "kube-api-access-89vp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:45.523483 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:45.523446    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b" (UID: "a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:27:45.602632 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:45.602558    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89vp2\" (UniqueName: \"kubernetes.io/projected/a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b-kube-api-access-89vp2\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:27:45.602632 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:45.602587    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b-model-cache\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:27:46.295777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:46.295742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-f2pml" event={"ID":"a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b","Type":"ContainerDied","Data":"e81ab43af7c9b2307b8ed59c23e0c890123b04feff9f2299d40106c37a7d0afd"}
Sep 11 16:27:46.295777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:46.295761    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-f2pml"
Sep 11 16:27:46.295777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:46.295784    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e81ab43af7c9b2307b8ed59c23e0c890123b04feff9f2299d40106c37a7d0afd"
Sep 11 16:27:59.158327 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.158284    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"]
Sep 11 16:27:59.158678 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.158622    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b" containerName="download-hf-model"
Sep 11 16:27:59.158678 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.158634    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b" containerName="download-hf-model"
Sep 11 16:27:59.158678 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.158653    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b" containerName="s3-tls-init-serving"
Sep 11 16:27:59.158678 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.158659    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b" containerName="s3-tls-init-serving"
Sep 11 16:27:59.158805 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.158731    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b" containerName="s3-tls-init-serving"
Sep 11 16:27:59.163017 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.162999    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.166190 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.166160    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:27:59.166320 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.166204    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-5d5066\""
Sep 11 16:27:59.166763 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.166653    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-5d5066-kube-rbac-proxy-sar-config\""
Sep 11 16:27:59.166763 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.166714    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:27:59.166763 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.166721    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-5d5066-predictor-serving-cert\""
Sep 11 16:27:59.166763 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.166724    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-5d5066-dockercfg-2jqg6\""
Sep 11 16:27:59.167069 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.166849    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:27:59.174389 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.174365    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"]
Sep 11 16:27:59.215081 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.215054    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kserve-provision-location\") pod \"isvc-init-fail-5d5066-predictor-576dcc8968-gkw85\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.215250 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.215088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-proxy-tls\") pod \"isvc-init-fail-5d5066-predictor-576dcc8968-gkw85\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.215250 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.215115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dtjm\" (UniqueName: \"kubernetes.io/projected/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kube-api-access-2dtjm\") pod \"isvc-init-fail-5d5066-predictor-576dcc8968-gkw85\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.215250 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.215214    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-5d5066-predictor-576dcc8968-gkw85\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.215250 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.215241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-cabundle-cert\") pod \"isvc-init-fail-5d5066-predictor-576dcc8968-gkw85\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.316422 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.316392    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kserve-provision-location\") pod \"isvc-init-fail-5d5066-predictor-576dcc8968-gkw85\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.316568 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.316432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-proxy-tls\") pod \"isvc-init-fail-5d5066-predictor-576dcc8968-gkw85\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.316568 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.316534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtjm\" (UniqueName: \"kubernetes.io/projected/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kube-api-access-2dtjm\") pod \"isvc-init-fail-5d5066-predictor-576dcc8968-gkw85\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.316568 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.316563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-5d5066-predictor-576dcc8968-gkw85\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.316696 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.316580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-cabundle-cert\") pod \"isvc-init-fail-5d5066-predictor-576dcc8968-gkw85\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.316853 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.316828    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kserve-provision-location\") pod \"isvc-init-fail-5d5066-predictor-576dcc8968-gkw85\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.317264 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.317242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-cabundle-cert\") pod \"isvc-init-fail-5d5066-predictor-576dcc8968-gkw85\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.317353 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.317267    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-5d5066-predictor-576dcc8968-gkw85\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.318869 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.318845    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-proxy-tls\") pod \"isvc-init-fail-5d5066-predictor-576dcc8968-gkw85\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.326435 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.326412    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dtjm\" (UniqueName: \"kubernetes.io/projected/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kube-api-access-2dtjm\") pod \"isvc-init-fail-5d5066-predictor-576dcc8968-gkw85\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") " pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.474413 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.474326    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:27:59.749155 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:27:59.749129    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"]
Sep 11 16:27:59.751276 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:27:59.751246    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8de051_cb8a_4a41_8ef9_831a9ddc154e.slice/crio-ae152ad5dd5120314c74247086590d0ef50d75dd7a3f44b4ef8ce30f941b8935 WatchSource:0}: Error finding container ae152ad5dd5120314c74247086590d0ef50d75dd7a3f44b4ef8ce30f941b8935: Status 404 returned error can't find the container with id ae152ad5dd5120314c74247086590d0ef50d75dd7a3f44b4ef8ce30f941b8935
Sep 11 16:28:00.355545 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:00.355506    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85" event={"ID":"1a8de051-cb8a-4a41-8ef9-831a9ddc154e","Type":"ContainerStarted","Data":"208c925a516c2c4addf67d5e484f617f9cad9b1eddecea73f3fa13919794c45e"}
Sep 11 16:28:00.356006 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:00.355564    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85" event={"ID":"1a8de051-cb8a-4a41-8ef9-831a9ddc154e","Type":"ContainerStarted","Data":"ae152ad5dd5120314c74247086590d0ef50d75dd7a3f44b4ef8ce30f941b8935"}
Sep 11 16:28:03.369962 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:03.369934    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5d5066-predictor-576dcc8968-gkw85_1a8de051-cb8a-4a41-8ef9-831a9ddc154e/storage-initializer/0.log"
Sep 11 16:28:03.370337 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:03.369974    2561 generic.go:356] "Generic (PLEG): container finished" podID="1a8de051-cb8a-4a41-8ef9-831a9ddc154e" containerID="208c925a516c2c4addf67d5e484f617f9cad9b1eddecea73f3fa13919794c45e" exitCode=1
Sep 11 16:28:03.370337 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:03.370002    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85" event={"ID":"1a8de051-cb8a-4a41-8ef9-831a9ddc154e","Type":"ContainerDied","Data":"208c925a516c2c4addf67d5e484f617f9cad9b1eddecea73f3fa13919794c45e"}
Sep 11 16:28:04.376546 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:04.376519    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5d5066-predictor-576dcc8968-gkw85_1a8de051-cb8a-4a41-8ef9-831a9ddc154e/storage-initializer/0.log"
Sep 11 16:28:04.376962 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:04.376596    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85" event={"ID":"1a8de051-cb8a-4a41-8ef9-831a9ddc154e","Type":"ContainerStarted","Data":"12ddbd5fb70671d14dd39580e3bd43312ad8da5681a87b1e6dd28370c9159ce0"}
Sep 11 16:28:07.397428 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:07.397351    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5d5066-predictor-576dcc8968-gkw85_1a8de051-cb8a-4a41-8ef9-831a9ddc154e/storage-initializer/1.log"
Sep 11 16:28:07.397832 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:07.397779    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5d5066-predictor-576dcc8968-gkw85_1a8de051-cb8a-4a41-8ef9-831a9ddc154e/storage-initializer/0.log"
Sep 11 16:28:07.397832 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:07.397819    2561 generic.go:356] "Generic (PLEG): container finished" podID="1a8de051-cb8a-4a41-8ef9-831a9ddc154e" containerID="12ddbd5fb70671d14dd39580e3bd43312ad8da5681a87b1e6dd28370c9159ce0" exitCode=1
Sep 11 16:28:07.397938 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:07.397890    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85" event={"ID":"1a8de051-cb8a-4a41-8ef9-831a9ddc154e","Type":"ContainerDied","Data":"12ddbd5fb70671d14dd39580e3bd43312ad8da5681a87b1e6dd28370c9159ce0"}
Sep 11 16:28:07.397992 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:07.397978    2561 scope.go:117] "RemoveContainer" containerID="208c925a516c2c4addf67d5e484f617f9cad9b1eddecea73f3fa13919794c45e"
Sep 11 16:28:07.398319 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:07.398293    2561 scope.go:117] "RemoveContainer" containerID="208c925a516c2c4addf67d5e484f617f9cad9b1eddecea73f3fa13919794c45e"
Sep 11 16:28:07.411295 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:28:07.410655    2561 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-5d5066-predictor-576dcc8968-gkw85_kserve-ci-e2e-test_1a8de051-cb8a-4a41-8ef9-831a9ddc154e_0 in pod sandbox ae152ad5dd5120314c74247086590d0ef50d75dd7a3f44b4ef8ce30f941b8935 from index: no such id: '208c925a516c2c4addf67d5e484f617f9cad9b1eddecea73f3fa13919794c45e'" containerID="208c925a516c2c4addf67d5e484f617f9cad9b1eddecea73f3fa13919794c45e"
Sep 11 16:28:07.411295 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:28:07.410702    2561 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-5d5066-predictor-576dcc8968-gkw85_kserve-ci-e2e-test_1a8de051-cb8a-4a41-8ef9-831a9ddc154e_0 in pod sandbox ae152ad5dd5120314c74247086590d0ef50d75dd7a3f44b4ef8ce30f941b8935 from index: no such id: '208c925a516c2c4addf67d5e484f617f9cad9b1eddecea73f3fa13919794c45e'; Skipping pod \"isvc-init-fail-5d5066-predictor-576dcc8968-gkw85_kserve-ci-e2e-test(1a8de051-cb8a-4a41-8ef9-831a9ddc154e)\"" logger="UnhandledError"
Sep 11 16:28:07.411295 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:28:07.410959    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-5d5066-predictor-576dcc8968-gkw85_kserve-ci-e2e-test(1a8de051-cb8a-4a41-8ef9-831a9ddc154e)\"" pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85" podUID="1a8de051-cb8a-4a41-8ef9-831a9ddc154e"
Sep 11 16:28:08.403768 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:08.403743    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5d5066-predictor-576dcc8968-gkw85_1a8de051-cb8a-4a41-8ef9-831a9ddc154e/storage-initializer/1.log"
Sep 11 16:28:08.665344 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:08.665265    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"]
Sep 11 16:28:08.788391 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:08.788370    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5d5066-predictor-576dcc8968-gkw85_1a8de051-cb8a-4a41-8ef9-831a9ddc154e/storage-initializer/1.log"
Sep 11 16:28:08.791592 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:08.791576    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:28:08.899992 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:08.899962    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dtjm\" (UniqueName: \"kubernetes.io/projected/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kube-api-access-2dtjm\") pod \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") "
Sep 11 16:28:08.900134 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:08.900012    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-proxy-tls\") pod \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") "
Sep 11 16:28:08.900134 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:08.900030    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kube-rbac-proxy-sar-config\") pod \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") "
Sep 11 16:28:08.900134 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:08.900053    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kserve-provision-location\") pod \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") "
Sep 11 16:28:08.900134 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:08.900119    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-cabundle-cert\") pod \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\" (UID: \"1a8de051-cb8a-4a41-8ef9-831a9ddc154e\") "
Sep 11 16:28:08.900402 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:08.900367    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a8de051-cb8a-4a41-8ef9-831a9ddc154e" (UID: "1a8de051-cb8a-4a41-8ef9-831a9ddc154e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:28:08.900480 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:08.900460    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a8de051-cb8a-4a41-8ef9-831a9ddc154e" (UID: "1a8de051-cb8a-4a41-8ef9-831a9ddc154e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:28:08.900587 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:08.900547    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a8de051-cb8a-4a41-8ef9-831a9ddc154e" (UID: "1a8de051-cb8a-4a41-8ef9-831a9ddc154e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:28:08.902116 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:08.902087    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a8de051-cb8a-4a41-8ef9-831a9ddc154e" (UID: "1a8de051-cb8a-4a41-8ef9-831a9ddc154e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:28:08.902216 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:08.902164    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kube-api-access-2dtjm" (OuterVolumeSpecName: "kube-api-access-2dtjm") pod "1a8de051-cb8a-4a41-8ef9-831a9ddc154e" (UID: "1a8de051-cb8a-4a41-8ef9-831a9ddc154e"). InnerVolumeSpecName "kube-api-access-2dtjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:28:09.001698 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:09.001638    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dtjm\" (UniqueName: \"kubernetes.io/projected/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kube-api-access-2dtjm\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:28:09.001698 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:09.001662    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:28:09.001698 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:09.001671    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:28:09.001698 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:09.001679    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:28:09.001698 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:09.001689    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8de051-cb8a-4a41-8ef9-831a9ddc154e-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:28:09.409097 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:09.409073    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-5d5066-predictor-576dcc8968-gkw85_1a8de051-cb8a-4a41-8ef9-831a9ddc154e/storage-initializer/1.log"
Sep 11 16:28:09.409470 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:09.409159    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85" event={"ID":"1a8de051-cb8a-4a41-8ef9-831a9ddc154e","Type":"ContainerDied","Data":"ae152ad5dd5120314c74247086590d0ef50d75dd7a3f44b4ef8ce30f941b8935"}
Sep 11 16:28:09.409470 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:09.409169    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"
Sep 11 16:28:09.409470 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:09.409237    2561 scope.go:117] "RemoveContainer" containerID="12ddbd5fb70671d14dd39580e3bd43312ad8da5681a87b1e6dd28370c9159ce0"
Sep 11 16:28:09.448454 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:09.448425    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"]
Sep 11 16:28:09.455426 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:09.455402    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-5d5066-predictor-576dcc8968-gkw85"]
Sep 11 16:28:11.320914 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:11.320883    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a8de051-cb8a-4a41-8ef9-831a9ddc154e" path="/var/lib/kubelet/pods/1a8de051-cb8a-4a41-8ef9-831a9ddc154e/volumes"
Sep 11 16:28:13.941046 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:13.941010    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"]
Sep 11 16:28:13.941550 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:13.941534    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8de051-cb8a-4a41-8ef9-831a9ddc154e" containerName="storage-initializer"
Sep 11 16:28:13.941594 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:13.941553    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8de051-cb8a-4a41-8ef9-831a9ddc154e" containerName="storage-initializer"
Sep 11 16:28:13.941594 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:13.941584    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8de051-cb8a-4a41-8ef9-831a9ddc154e" containerName="storage-initializer"
Sep 11 16:28:13.941594 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:13.941592    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8de051-cb8a-4a41-8ef9-831a9ddc154e" containerName="storage-initializer"
Sep 11 16:28:13.941686 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:13.941663    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8de051-cb8a-4a41-8ef9-831a9ddc154e" containerName="storage-initializer"
Sep 11 16:28:13.941686 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:13.941678    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8de051-cb8a-4a41-8ef9-831a9ddc154e" containerName="storage-initializer"
Sep 11 16:28:13.946922 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:13.946897    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:13.949292 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:13.949273    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 11 16:28:13.949554 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:13.949538    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 11 16:28:13.949881 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:13.949856    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:28:13.950073 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:13.949879    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 16:28:13.950073 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:13.949863    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:28:13.950073 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:13.949943    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:28:13.950073 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:13.949972    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:28:13.956054 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:13.955835    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"]
Sep 11 16:28:14.045394 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.045352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/644b0c50-e8a8-47d2-96bb-24c72138d6ab-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mllcs\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:14.045394 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.045396    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/644b0c50-e8a8-47d2-96bb-24c72138d6ab-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mllcs\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:14.045623 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.045454    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxdrf\" (UniqueName: \"kubernetes.io/projected/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kube-api-access-sxdrf\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mllcs\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:14.045623 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.045528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mllcs\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:14.045623 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.045581    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mllcs\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:14.146274 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.146237    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/644b0c50-e8a8-47d2-96bb-24c72138d6ab-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mllcs\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:14.146274 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.146270    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/644b0c50-e8a8-47d2-96bb-24c72138d6ab-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mllcs\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:14.146539 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.146297    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxdrf\" (UniqueName: \"kubernetes.io/projected/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kube-api-access-sxdrf\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mllcs\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:14.146539 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.146334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mllcs\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:14.146539 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.146351    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mllcs\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:14.146834 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.146809    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mllcs\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:14.147063 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.147040    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/644b0c50-e8a8-47d2-96bb-24c72138d6ab-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mllcs\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:14.147121 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.147066    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mllcs\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:14.148707 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.148679    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/644b0c50-e8a8-47d2-96bb-24c72138d6ab-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mllcs\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:14.155688 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.155666    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxdrf\" (UniqueName: \"kubernetes.io/projected/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kube-api-access-sxdrf\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mllcs\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:14.260079 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.259986    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:14.536201 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:14.536160    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"]
Sep 11 16:28:14.538906 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:28:14.538880    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644b0c50_e8a8_47d2_96bb_24c72138d6ab.slice/crio-c097c321345fcd10afce1ff0d0fac14c1b630f06ef4a83fa52c1d731c5c3e217 WatchSource:0}: Error finding container c097c321345fcd10afce1ff0d0fac14c1b630f06ef4a83fa52c1d731c5c3e217: Status 404 returned error can't find the container with id c097c321345fcd10afce1ff0d0fac14c1b630f06ef4a83fa52c1d731c5c3e217
Sep 11 16:28:15.438411 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:15.438374    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" event={"ID":"644b0c50-e8a8-47d2-96bb-24c72138d6ab","Type":"ContainerStarted","Data":"098c75a4d8a11c920dc397477d9e93e51c18d074a48b14035bfd63308d0ef3cc"}
Sep 11 16:28:15.438958 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:15.438429    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" event={"ID":"644b0c50-e8a8-47d2-96bb-24c72138d6ab","Type":"ContainerStarted","Data":"c097c321345fcd10afce1ff0d0fac14c1b630f06ef4a83fa52c1d731c5c3e217"}
Sep 11 16:28:18.453348 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:18.453314    2561 generic.go:356] "Generic (PLEG): container finished" podID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerID="098c75a4d8a11c920dc397477d9e93e51c18d074a48b14035bfd63308d0ef3cc" exitCode=0
Sep 11 16:28:18.453734 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:18.453386    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" event={"ID":"644b0c50-e8a8-47d2-96bb-24c72138d6ab","Type":"ContainerDied","Data":"098c75a4d8a11c920dc397477d9e93e51c18d074a48b14035bfd63308d0ef3cc"}
Sep 11 16:28:42.601644 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:42.601558    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" event={"ID":"644b0c50-e8a8-47d2-96bb-24c72138d6ab","Type":"ContainerStarted","Data":"5502108f4d96610ce59b3ebf382c409362af9924de3f75a8798a7e5f72782366"}
Sep 11 16:28:45.619185 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:45.619132    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" event={"ID":"644b0c50-e8a8-47d2-96bb-24c72138d6ab","Type":"ContainerStarted","Data":"c4d0575e1a0ff8e1ee2ed446a7a9c061ab42d9b918c4a7e2991cf851a69f495e"}
Sep 11 16:28:45.619604 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:45.619229    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:45.619604 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:45.619260    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:45.620847 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:45.620818    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 11 16:28:45.639730 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:45.639653    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" podStartSLOduration=2.863959743 podStartE2EDuration="32.639636802s" totalImagesPullingTime="26.082720661s" totalInitContainerRuntime="3.692956398s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:28:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:28:45.63816317 +0000 UTC m=+730.978257216" watchObservedRunningTime="2026-09-11 16:28:45.639636802 +0000 UTC m=+730.979730848"
Sep 11 16:28:46.623969 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:46.623931    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 11 16:28:51.628339 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:51.628305    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:28:51.628997 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:28:51.628969    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 11 16:29:01.629489 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:29:01.629448    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 11 16:29:11.629377 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:29:11.629329    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 11 16:29:21.629283 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:29:21.629223    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 11 16:29:31.629484 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:29:31.629440    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 11 16:29:41.628932 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:29:41.628884    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 11 16:29:51.629472 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:29:51.629429    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 11 16:30:01.629385 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:01.629308    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:30:08.020936 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:08.020903    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"]
Sep 11 16:30:08.021349 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:08.021141    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kserve-container" containerID="cri-o://5502108f4d96610ce59b3ebf382c409362af9924de3f75a8798a7e5f72782366" gracePeriod=30
Sep 11 16:30:08.021349 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:08.021204    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kube-rbac-proxy" containerID="cri-o://c4d0575e1a0ff8e1ee2ed446a7a9c061ab42d9b918c4a7e2991cf851a69f495e" gracePeriod=30
Sep 11 16:30:09.018365 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:09.018327    2561 generic.go:356] "Generic (PLEG): container finished" podID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerID="c4d0575e1a0ff8e1ee2ed446a7a9c061ab42d9b918c4a7e2991cf851a69f495e" exitCode=2
Sep 11 16:30:09.018528 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:09.018399    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" event={"ID":"644b0c50-e8a8-47d2-96bb-24c72138d6ab","Type":"ContainerDied","Data":"c4d0575e1a0ff8e1ee2ed446a7a9c061ab42d9b918c4a7e2991cf851a69f495e"}
Sep 11 16:30:11.625044 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:11.624998    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 11 16:30:11.629889 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:11.629859    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 11 16:30:12.761923 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:12.761898    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:30:12.805170 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:12.805140    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kube-rbac-proxy-sar-config\") pod \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") "
Sep 11 16:30:12.805346 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:12.805241    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/644b0c50-e8a8-47d2-96bb-24c72138d6ab-cabundle-cert\") pod \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") "
Sep 11 16:30:12.805346 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:12.805282    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxdrf\" (UniqueName: \"kubernetes.io/projected/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kube-api-access-sxdrf\") pod \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") "
Sep 11 16:30:12.805346 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:12.805334    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kserve-provision-location\") pod \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") "
Sep 11 16:30:12.805485 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:12.805372    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/644b0c50-e8a8-47d2-96bb-24c72138d6ab-proxy-tls\") pod \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\" (UID: \"644b0c50-e8a8-47d2-96bb-24c72138d6ab\") "
Sep 11 16:30:12.805686 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:12.805654    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "644b0c50-e8a8-47d2-96bb-24c72138d6ab" (UID: "644b0c50-e8a8-47d2-96bb-24c72138d6ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:30:12.805686 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:12.805654    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "644b0c50-e8a8-47d2-96bb-24c72138d6ab" (UID: "644b0c50-e8a8-47d2-96bb-24c72138d6ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:30:12.805837 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:12.805654    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644b0c50-e8a8-47d2-96bb-24c72138d6ab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "644b0c50-e8a8-47d2-96bb-24c72138d6ab" (UID: "644b0c50-e8a8-47d2-96bb-24c72138d6ab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:30:12.807474 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:12.807438    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kube-api-access-sxdrf" (OuterVolumeSpecName: "kube-api-access-sxdrf") pod "644b0c50-e8a8-47d2-96bb-24c72138d6ab" (UID: "644b0c50-e8a8-47d2-96bb-24c72138d6ab"). InnerVolumeSpecName "kube-api-access-sxdrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:30:12.807555 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:12.807442    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644b0c50-e8a8-47d2-96bb-24c72138d6ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "644b0c50-e8a8-47d2-96bb-24c72138d6ab" (UID: "644b0c50-e8a8-47d2-96bb-24c72138d6ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:30:12.906074 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:12.906033    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/644b0c50-e8a8-47d2-96bb-24c72138d6ab-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:30:12.906074 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:12.906069    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxdrf\" (UniqueName: \"kubernetes.io/projected/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kube-api-access-sxdrf\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:30:12.906074 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:12.906080    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:30:12.906323 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:12.906090    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/644b0c50-e8a8-47d2-96bb-24c72138d6ab-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:30:12.906323 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:12.906099    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/644b0c50-e8a8-47d2-96bb-24c72138d6ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:30:13.040251 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.040139    2561 generic.go:356] "Generic (PLEG): container finished" podID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerID="5502108f4d96610ce59b3ebf382c409362af9924de3f75a8798a7e5f72782366" exitCode=0
Sep 11 16:30:13.040251 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.040213    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"
Sep 11 16:30:13.040251 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.040225    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" event={"ID":"644b0c50-e8a8-47d2-96bb-24c72138d6ab","Type":"ContainerDied","Data":"5502108f4d96610ce59b3ebf382c409362af9924de3f75a8798a7e5f72782366"}
Sep 11 16:30:13.040528 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.040302    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs" event={"ID":"644b0c50-e8a8-47d2-96bb-24c72138d6ab","Type":"ContainerDied","Data":"c097c321345fcd10afce1ff0d0fac14c1b630f06ef4a83fa52c1d731c5c3e217"}
Sep 11 16:30:13.040528 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.040341    2561 scope.go:117] "RemoveContainer" containerID="c4d0575e1a0ff8e1ee2ed446a7a9c061ab42d9b918c4a7e2991cf851a69f495e"
Sep 11 16:30:13.049103 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.049082    2561 scope.go:117] "RemoveContainer" containerID="5502108f4d96610ce59b3ebf382c409362af9924de3f75a8798a7e5f72782366"
Sep 11 16:30:13.057054 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.057032    2561 scope.go:117] "RemoveContainer" containerID="098c75a4d8a11c920dc397477d9e93e51c18d074a48b14035bfd63308d0ef3cc"
Sep 11 16:30:13.064524 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.064502    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"]
Sep 11 16:30:13.065166 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.065145    2561 scope.go:117] "RemoveContainer" containerID="c4d0575e1a0ff8e1ee2ed446a7a9c061ab42d9b918c4a7e2991cf851a69f495e"
Sep 11 16:30:13.065481 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:30:13.065463    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d0575e1a0ff8e1ee2ed446a7a9c061ab42d9b918c4a7e2991cf851a69f495e\": container with ID starting with c4d0575e1a0ff8e1ee2ed446a7a9c061ab42d9b918c4a7e2991cf851a69f495e not found: ID does not exist" containerID="c4d0575e1a0ff8e1ee2ed446a7a9c061ab42d9b918c4a7e2991cf851a69f495e"
Sep 11 16:30:13.065535 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.065490    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d0575e1a0ff8e1ee2ed446a7a9c061ab42d9b918c4a7e2991cf851a69f495e"} err="failed to get container status \"c4d0575e1a0ff8e1ee2ed446a7a9c061ab42d9b918c4a7e2991cf851a69f495e\": rpc error: code = NotFound desc = could not find container \"c4d0575e1a0ff8e1ee2ed446a7a9c061ab42d9b918c4a7e2991cf851a69f495e\": container with ID starting with c4d0575e1a0ff8e1ee2ed446a7a9c061ab42d9b918c4a7e2991cf851a69f495e not found: ID does not exist"
Sep 11 16:30:13.065535 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.065508    2561 scope.go:117] "RemoveContainer" containerID="5502108f4d96610ce59b3ebf382c409362af9924de3f75a8798a7e5f72782366"
Sep 11 16:30:13.065796 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:30:13.065771    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5502108f4d96610ce59b3ebf382c409362af9924de3f75a8798a7e5f72782366\": container with ID starting with 5502108f4d96610ce59b3ebf382c409362af9924de3f75a8798a7e5f72782366 not found: ID does not exist" containerID="5502108f4d96610ce59b3ebf382c409362af9924de3f75a8798a7e5f72782366"
Sep 11 16:30:13.065835 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.065802    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5502108f4d96610ce59b3ebf382c409362af9924de3f75a8798a7e5f72782366"} err="failed to get container status \"5502108f4d96610ce59b3ebf382c409362af9924de3f75a8798a7e5f72782366\": rpc error: code = NotFound desc = could not find container \"5502108f4d96610ce59b3ebf382c409362af9924de3f75a8798a7e5f72782366\": container with ID starting with 5502108f4d96610ce59b3ebf382c409362af9924de3f75a8798a7e5f72782366 not found: ID does not exist"
Sep 11 16:30:13.065835 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.065820    2561 scope.go:117] "RemoveContainer" containerID="098c75a4d8a11c920dc397477d9e93e51c18d074a48b14035bfd63308d0ef3cc"
Sep 11 16:30:13.066072 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:30:13.066054    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"098c75a4d8a11c920dc397477d9e93e51c18d074a48b14035bfd63308d0ef3cc\": container with ID starting with 098c75a4d8a11c920dc397477d9e93e51c18d074a48b14035bfd63308d0ef3cc not found: ID does not exist" containerID="098c75a4d8a11c920dc397477d9e93e51c18d074a48b14035bfd63308d0ef3cc"
Sep 11 16:30:13.066131 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.066080    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"098c75a4d8a11c920dc397477d9e93e51c18d074a48b14035bfd63308d0ef3cc"} err="failed to get container status \"098c75a4d8a11c920dc397477d9e93e51c18d074a48b14035bfd63308d0ef3cc\": rpc error: code = NotFound desc = could not find container \"098c75a4d8a11c920dc397477d9e93e51c18d074a48b14035bfd63308d0ef3cc\": container with ID starting with 098c75a4d8a11c920dc397477d9e93e51c18d074a48b14035bfd63308d0ef3cc not found: ID does not exist"
Sep 11 16:30:13.072858 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.072835    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mllcs"]
Sep 11 16:30:13.223224 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.223187    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"]
Sep 11 16:30:13.223623 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.223605    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kube-rbac-proxy"
Sep 11 16:30:13.223690 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.223627    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kube-rbac-proxy"
Sep 11 16:30:13.223690 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.223677    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="storage-initializer"
Sep 11 16:30:13.223690 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.223689    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="storage-initializer"
Sep 11 16:30:13.223836 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.223711    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kserve-container"
Sep 11 16:30:13.223836 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.223721    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kserve-container"
Sep 11 16:30:13.223836 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.223800    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kube-rbac-proxy"
Sep 11 16:30:13.223836 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.223812    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" containerName="kserve-container"
Sep 11 16:30:13.228416 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.228401    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.230796 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.230773    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 11 16:30:13.231020 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.230998    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 16:30:13.231158 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.231064    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:30:13.231245 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.231222    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 11 16:30:13.231299 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.231255    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:30:13.231378 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.231346    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:30:13.231611 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.231596    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:30:13.240226 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.240204    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"]
Sep 11 16:30:13.310122 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.310038    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48e50cd6-7e7b-4211-907d-c418493d15dd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-rv59q\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.310122 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.310080    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48e50cd6-7e7b-4211-907d-c418493d15dd-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-rv59q\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.310307 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.310138    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48e50cd6-7e7b-4211-907d-c418493d15dd-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-rv59q\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.310307 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.310154    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7ck8\" (UniqueName: \"kubernetes.io/projected/48e50cd6-7e7b-4211-907d-c418493d15dd-kube-api-access-j7ck8\") pod \"isvc-predictive-xgboost-predictor-695f978978-rv59q\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.310307 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.310211    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48e50cd6-7e7b-4211-907d-c418493d15dd-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-rv59q\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.321806 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.321765    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="644b0c50-e8a8-47d2-96bb-24c72138d6ab" path="/var/lib/kubelet/pods/644b0c50-e8a8-47d2-96bb-24c72138d6ab/volumes"
Sep 11 16:30:13.411637 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.411597    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48e50cd6-7e7b-4211-907d-c418493d15dd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-rv59q\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.411811 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.411642    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48e50cd6-7e7b-4211-907d-c418493d15dd-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-rv59q\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.411811 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.411686    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48e50cd6-7e7b-4211-907d-c418493d15dd-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-rv59q\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.411811 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.411713    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7ck8\" (UniqueName: \"kubernetes.io/projected/48e50cd6-7e7b-4211-907d-c418493d15dd-kube-api-access-j7ck8\") pod \"isvc-predictive-xgboost-predictor-695f978978-rv59q\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.411811 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.411750    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48e50cd6-7e7b-4211-907d-c418493d15dd-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-rv59q\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.412280 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.412244    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48e50cd6-7e7b-4211-907d-c418493d15dd-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-rv59q\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.412506 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.412479    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48e50cd6-7e7b-4211-907d-c418493d15dd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-rv59q\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.412506 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.412492    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48e50cd6-7e7b-4211-907d-c418493d15dd-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-rv59q\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.414214 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.414195    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48e50cd6-7e7b-4211-907d-c418493d15dd-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-rv59q\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.421850 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.421828    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7ck8\" (UniqueName: \"kubernetes.io/projected/48e50cd6-7e7b-4211-907d-c418493d15dd-kube-api-access-j7ck8\") pod \"isvc-predictive-xgboost-predictor-695f978978-rv59q\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.539805 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.539767    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:13.815439 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:13.815406    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"]
Sep 11 16:30:13.817625 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:30:13.817595    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e50cd6_7e7b_4211_907d_c418493d15dd.slice/crio-26db9cc2cd12528c7b7e7cbcd4c01525dcaf89ec90ef67d5ce80f3a160082f27 WatchSource:0}: Error finding container 26db9cc2cd12528c7b7e7cbcd4c01525dcaf89ec90ef67d5ce80f3a160082f27: Status 404 returned error can't find the container with id 26db9cc2cd12528c7b7e7cbcd4c01525dcaf89ec90ef67d5ce80f3a160082f27
Sep 11 16:30:14.056701 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:14.056657    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" event={"ID":"48e50cd6-7e7b-4211-907d-c418493d15dd","Type":"ContainerStarted","Data":"456d9c47145c792abe21826b68d042ea217cf0dc134e9b4e56832af18980bcfd"}
Sep 11 16:30:14.056864 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:14.056775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" event={"ID":"48e50cd6-7e7b-4211-907d-c418493d15dd","Type":"ContainerStarted","Data":"26db9cc2cd12528c7b7e7cbcd4c01525dcaf89ec90ef67d5ce80f3a160082f27"}
Sep 11 16:30:18.078488 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:18.078405    2561 generic.go:356] "Generic (PLEG): container finished" podID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerID="456d9c47145c792abe21826b68d042ea217cf0dc134e9b4e56832af18980bcfd" exitCode=0
Sep 11 16:30:18.078488 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:18.078475    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" event={"ID":"48e50cd6-7e7b-4211-907d-c418493d15dd","Type":"ContainerDied","Data":"456d9c47145c792abe21826b68d042ea217cf0dc134e9b4e56832af18980bcfd"}
Sep 11 16:30:19.086383 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:19.086351    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" event={"ID":"48e50cd6-7e7b-4211-907d-c418493d15dd","Type":"ContainerStarted","Data":"bd16cba906f3002a80da5257979d8a1925e2eeb27ccf5889b1e5a75296facb65"}
Sep 11 16:30:19.086757 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:19.086414    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" event={"ID":"48e50cd6-7e7b-4211-907d-c418493d15dd","Type":"ContainerStarted","Data":"fbe655df393dcdffa2abb05f7909cddac3f3f69a9373cd5d9a55452989739478"}
Sep 11 16:30:19.086757 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:19.086635    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:19.086842 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:19.086774    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:19.087962 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:19.087938    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 11 16:30:19.112896 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:19.112840    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" podStartSLOduration=2.335519046 podStartE2EDuration="6.112821933s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.777302887s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:30:19.112667581 +0000 UTC m=+824.452761630" watchObservedRunningTime="2026-09-11 16:30:19.112821933 +0000 UTC m=+824.452915981"
Sep 11 16:30:20.091052 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:20.091000    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 11 16:30:25.096119 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:25.096089    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:30:25.096614 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:25.096587    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 11 16:30:35.096546 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:35.096508    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 11 16:30:45.096711 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:45.096673    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 11 16:30:55.096755 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:30:55.096704    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 11 16:31:05.097379 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:05.097335    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 11 16:31:15.096986 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:15.096940    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 11 16:31:25.096853 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:25.096813    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 11 16:31:35.097740 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:35.097710    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:31:35.172571 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:35.172542    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:31:35.300488 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:35.300462    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:31:43.314456 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:43.314425    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"]
Sep 11 16:31:43.314875 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:43.314645    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kserve-container" containerID="cri-o://fbe655df393dcdffa2abb05f7909cddac3f3f69a9373cd5d9a55452989739478" gracePeriod=30
Sep 11 16:31:43.314875 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:43.314664    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kube-rbac-proxy" containerID="cri-o://bd16cba906f3002a80da5257979d8a1925e2eeb27ccf5889b1e5a75296facb65" gracePeriod=30
Sep 11 16:31:43.487169 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:43.487139    2561 generic.go:356] "Generic (PLEG): container finished" podID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerID="bd16cba906f3002a80da5257979d8a1925e2eeb27ccf5889b1e5a75296facb65" exitCode=2
Sep 11 16:31:43.487346 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:43.487206    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" event={"ID":"48e50cd6-7e7b-4211-907d-c418493d15dd","Type":"ContainerDied","Data":"bd16cba906f3002a80da5257979d8a1925e2eeb27ccf5889b1e5a75296facb65"}
Sep 11 16:31:45.091988 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:45.091950    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 11 16:31:45.097381 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:45.097360    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 11 16:31:47.857138 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:47.857112    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:31:47.938671 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:47.938646    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48e50cd6-7e7b-4211-907d-c418493d15dd-kube-rbac-proxy-sar-config\") pod \"48e50cd6-7e7b-4211-907d-c418493d15dd\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") "
Sep 11 16:31:47.938813 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:47.938680    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48e50cd6-7e7b-4211-907d-c418493d15dd-proxy-tls\") pod \"48e50cd6-7e7b-4211-907d-c418493d15dd\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") "
Sep 11 16:31:47.938813 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:47.938711    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7ck8\" (UniqueName: \"kubernetes.io/projected/48e50cd6-7e7b-4211-907d-c418493d15dd-kube-api-access-j7ck8\") pod \"48e50cd6-7e7b-4211-907d-c418493d15dd\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") "
Sep 11 16:31:47.938813 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:47.938757    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48e50cd6-7e7b-4211-907d-c418493d15dd-kserve-provision-location\") pod \"48e50cd6-7e7b-4211-907d-c418493d15dd\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") "
Sep 11 16:31:47.938813 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:47.938806    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48e50cd6-7e7b-4211-907d-c418493d15dd-cabundle-cert\") pod \"48e50cd6-7e7b-4211-907d-c418493d15dd\" (UID: \"48e50cd6-7e7b-4211-907d-c418493d15dd\") "
Sep 11 16:31:47.939143 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:47.939108    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e50cd6-7e7b-4211-907d-c418493d15dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48e50cd6-7e7b-4211-907d-c418493d15dd" (UID: "48e50cd6-7e7b-4211-907d-c418493d15dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:31:47.939284 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:47.939137    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e50cd6-7e7b-4211-907d-c418493d15dd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "48e50cd6-7e7b-4211-907d-c418493d15dd" (UID: "48e50cd6-7e7b-4211-907d-c418493d15dd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:31:47.939284 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:47.939231    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e50cd6-7e7b-4211-907d-c418493d15dd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "48e50cd6-7e7b-4211-907d-c418493d15dd" (UID: "48e50cd6-7e7b-4211-907d-c418493d15dd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:31:47.940902 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:47.940877    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e50cd6-7e7b-4211-907d-c418493d15dd-kube-api-access-j7ck8" (OuterVolumeSpecName: "kube-api-access-j7ck8") pod "48e50cd6-7e7b-4211-907d-c418493d15dd" (UID: "48e50cd6-7e7b-4211-907d-c418493d15dd"). InnerVolumeSpecName "kube-api-access-j7ck8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:31:47.940983 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:47.940878    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48e50cd6-7e7b-4211-907d-c418493d15dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "48e50cd6-7e7b-4211-907d-c418493d15dd" (UID: "48e50cd6-7e7b-4211-907d-c418493d15dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:31:48.039544 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.039516    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48e50cd6-7e7b-4211-907d-c418493d15dd-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:31:48.039544 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.039540    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48e50cd6-7e7b-4211-907d-c418493d15dd-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:31:48.039544 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.039549    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48e50cd6-7e7b-4211-907d-c418493d15dd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:31:48.039801 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.039558    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48e50cd6-7e7b-4211-907d-c418493d15dd-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:31:48.039801 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.039566    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7ck8\" (UniqueName: \"kubernetes.io/projected/48e50cd6-7e7b-4211-907d-c418493d15dd-kube-api-access-j7ck8\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:31:48.511579 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.511548    2561 generic.go:356] "Generic (PLEG): container finished" podID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerID="fbe655df393dcdffa2abb05f7909cddac3f3f69a9373cd5d9a55452989739478" exitCode=0
Sep 11 16:31:48.511750 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.511602    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"
Sep 11 16:31:48.511750 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.511625    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" event={"ID":"48e50cd6-7e7b-4211-907d-c418493d15dd","Type":"ContainerDied","Data":"fbe655df393dcdffa2abb05f7909cddac3f3f69a9373cd5d9a55452989739478"}
Sep 11 16:31:48.511750 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.511694    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q" event={"ID":"48e50cd6-7e7b-4211-907d-c418493d15dd","Type":"ContainerDied","Data":"26db9cc2cd12528c7b7e7cbcd4c01525dcaf89ec90ef67d5ce80f3a160082f27"}
Sep 11 16:31:48.511750 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.511721    2561 scope.go:117] "RemoveContainer" containerID="bd16cba906f3002a80da5257979d8a1925e2eeb27ccf5889b1e5a75296facb65"
Sep 11 16:31:48.521064 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.521045    2561 scope.go:117] "RemoveContainer" containerID="fbe655df393dcdffa2abb05f7909cddac3f3f69a9373cd5d9a55452989739478"
Sep 11 16:31:48.529324 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.529302    2561 scope.go:117] "RemoveContainer" containerID="456d9c47145c792abe21826b68d042ea217cf0dc134e9b4e56832af18980bcfd"
Sep 11 16:31:48.537032 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.537014    2561 scope.go:117] "RemoveContainer" containerID="bd16cba906f3002a80da5257979d8a1925e2eeb27ccf5889b1e5a75296facb65"
Sep 11 16:31:48.537346 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:31:48.537323    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd16cba906f3002a80da5257979d8a1925e2eeb27ccf5889b1e5a75296facb65\": container with ID starting with bd16cba906f3002a80da5257979d8a1925e2eeb27ccf5889b1e5a75296facb65 not found: ID does not exist" containerID="bd16cba906f3002a80da5257979d8a1925e2eeb27ccf5889b1e5a75296facb65"
Sep 11 16:31:48.537429 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.537350    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd16cba906f3002a80da5257979d8a1925e2eeb27ccf5889b1e5a75296facb65"} err="failed to get container status \"bd16cba906f3002a80da5257979d8a1925e2eeb27ccf5889b1e5a75296facb65\": rpc error: code = NotFound desc = could not find container \"bd16cba906f3002a80da5257979d8a1925e2eeb27ccf5889b1e5a75296facb65\": container with ID starting with bd16cba906f3002a80da5257979d8a1925e2eeb27ccf5889b1e5a75296facb65 not found: ID does not exist"
Sep 11 16:31:48.537429 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.537368    2561 scope.go:117] "RemoveContainer" containerID="fbe655df393dcdffa2abb05f7909cddac3f3f69a9373cd5d9a55452989739478"
Sep 11 16:31:48.537629 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:31:48.537611    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe655df393dcdffa2abb05f7909cddac3f3f69a9373cd5d9a55452989739478\": container with ID starting with fbe655df393dcdffa2abb05f7909cddac3f3f69a9373cd5d9a55452989739478 not found: ID does not exist" containerID="fbe655df393dcdffa2abb05f7909cddac3f3f69a9373cd5d9a55452989739478"
Sep 11 16:31:48.537665 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.537635    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe655df393dcdffa2abb05f7909cddac3f3f69a9373cd5d9a55452989739478"} err="failed to get container status \"fbe655df393dcdffa2abb05f7909cddac3f3f69a9373cd5d9a55452989739478\": rpc error: code = NotFound desc = could not find container \"fbe655df393dcdffa2abb05f7909cddac3f3f69a9373cd5d9a55452989739478\": container with ID starting with fbe655df393dcdffa2abb05f7909cddac3f3f69a9373cd5d9a55452989739478 not found: ID does not exist"
Sep 11 16:31:48.537665 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.537650    2561 scope.go:117] "RemoveContainer" containerID="456d9c47145c792abe21826b68d042ea217cf0dc134e9b4e56832af18980bcfd"
Sep 11 16:31:48.537901 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:31:48.537884    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456d9c47145c792abe21826b68d042ea217cf0dc134e9b4e56832af18980bcfd\": container with ID starting with 456d9c47145c792abe21826b68d042ea217cf0dc134e9b4e56832af18980bcfd not found: ID does not exist" containerID="456d9c47145c792abe21826b68d042ea217cf0dc134e9b4e56832af18980bcfd"
Sep 11 16:31:48.537941 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.537905    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456d9c47145c792abe21826b68d042ea217cf0dc134e9b4e56832af18980bcfd"} err="failed to get container status \"456d9c47145c792abe21826b68d042ea217cf0dc134e9b4e56832af18980bcfd\": rpc error: code = NotFound desc = could not find container \"456d9c47145c792abe21826b68d042ea217cf0dc134e9b4e56832af18980bcfd\": container with ID starting with 456d9c47145c792abe21826b68d042ea217cf0dc134e9b4e56832af18980bcfd not found: ID does not exist"
Sep 11 16:31:48.546262 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.546241    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"]
Sep 11 16:31:48.548809 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:48.548785    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-rv59q"]
Sep 11 16:31:49.321570 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:31:49.321539    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" path="/var/lib/kubelet/pods/48e50cd6-7e7b-4211-907d-c418493d15dd/volumes"
Sep 11 16:33:24.528189 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.528130    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"]
Sep 11 16:33:24.528754 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.528709    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kserve-container"
Sep 11 16:33:24.528754 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.528731    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kserve-container"
Sep 11 16:33:24.528754 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.528747    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kube-rbac-proxy"
Sep 11 16:33:24.528754 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.528755    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kube-rbac-proxy"
Sep 11 16:33:24.529036 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.528765    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="storage-initializer"
Sep 11 16:33:24.529036 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.528774    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="storage-initializer"
Sep 11 16:33:24.529036 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.528857    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kserve-container"
Sep 11 16:33:24.529036 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.528873    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="48e50cd6-7e7b-4211-907d-c418493d15dd" containerName="kube-rbac-proxy"
Sep 11 16:33:24.532283 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.532260    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:24.538258 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.538236    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:33:24.538390 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.538356    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:33:24.538928 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.538782    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 11 16:33:24.538928 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.538816    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 16:33:24.538928 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.538825    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:33:24.538928 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.538906    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:33:24.539146 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.539043    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 11 16:33:24.548542 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.548515    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"]
Sep 11 16:33:24.660810 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.660771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5b18639-c0d6-463e-a598-e3d60a378475-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-kvpvl\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:24.660810 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.660817    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckszd\" (UniqueName: \"kubernetes.io/projected/b5b18639-c0d6-463e-a598-e3d60a378475-kube-api-access-ckszd\") pod \"isvc-autogluon-v1-predictor-574b5b7495-kvpvl\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:24.661074 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.660863    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5b18639-c0d6-463e-a598-e3d60a378475-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-kvpvl\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:24.661074 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.660925    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5b18639-c0d6-463e-a598-e3d60a378475-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-kvpvl\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:24.661074 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.660954    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5b18639-c0d6-463e-a598-e3d60a378475-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-kvpvl\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:24.762394 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.762359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5b18639-c0d6-463e-a598-e3d60a378475-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-kvpvl\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:24.762394 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.762404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5b18639-c0d6-463e-a598-e3d60a378475-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-kvpvl\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:24.762633 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.762427    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5b18639-c0d6-463e-a598-e3d60a378475-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-kvpvl\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:24.762633 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.762514    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5b18639-c0d6-463e-a598-e3d60a378475-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-kvpvl\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:24.762633 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.762565    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckszd\" (UniqueName: \"kubernetes.io/projected/b5b18639-c0d6-463e-a598-e3d60a378475-kube-api-access-ckszd\") pod \"isvc-autogluon-v1-predictor-574b5b7495-kvpvl\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:24.762878 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.762851    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5b18639-c0d6-463e-a598-e3d60a378475-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-kvpvl\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:24.763146 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.763121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5b18639-c0d6-463e-a598-e3d60a378475-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-kvpvl\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:24.763146 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.763147    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5b18639-c0d6-463e-a598-e3d60a378475-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-kvpvl\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:24.764937 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.764915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5b18639-c0d6-463e-a598-e3d60a378475-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-kvpvl\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:24.773116 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.773092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckszd\" (UniqueName: \"kubernetes.io/projected/b5b18639-c0d6-463e-a598-e3d60a378475-kube-api-access-ckszd\") pod \"isvc-autogluon-v1-predictor-574b5b7495-kvpvl\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:24.842400 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:24.842305    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:33:25.124359 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:25.124299    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"]
Sep 11 16:33:25.127079 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:33:25.127048    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b18639_c0d6_463e_a598_e3d60a378475.slice/crio-55f8cae37fed1d5c9899121f2a42eb8d4be469bd07626e7fc1d423e0dbd4c9a3 WatchSource:0}: Error finding container 55f8cae37fed1d5c9899121f2a42eb8d4be469bd07626e7fc1d423e0dbd4c9a3: Status 404 returned error can't find the container with id 55f8cae37fed1d5c9899121f2a42eb8d4be469bd07626e7fc1d423e0dbd4c9a3
Sep 11 16:33:25.954083 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:25.954046    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl" event={"ID":"b5b18639-c0d6-463e-a598-e3d60a378475","Type":"ContainerStarted","Data":"7e005ae6fdec41364cd55095d39a23f0fdbdf91167a0f29376ebcd3bf370d2f6"}
Sep 11 16:33:25.954561 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:25.954155    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl" event={"ID":"b5b18639-c0d6-463e-a598-e3d60a378475","Type":"ContainerStarted","Data":"55f8cae37fed1d5c9899121f2a42eb8d4be469bd07626e7fc1d423e0dbd4c9a3"}
Sep 11 16:33:56.100049 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:56.100011    2561 generic.go:356] "Generic (PLEG): container finished" podID="b5b18639-c0d6-463e-a598-e3d60a378475" containerID="7e005ae6fdec41364cd55095d39a23f0fdbdf91167a0f29376ebcd3bf370d2f6" exitCode=0
Sep 11 16:33:56.100547 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:56.100088    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl" event={"ID":"b5b18639-c0d6-463e-a598-e3d60a378475","Type":"ContainerDied","Data":"7e005ae6fdec41364cd55095d39a23f0fdbdf91167a0f29376ebcd3bf370d2f6"}
Sep 11 16:33:56.101422 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:33:56.101405    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:35:58.768245 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:35:58.768132    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl" event={"ID":"b5b18639-c0d6-463e-a598-e3d60a378475","Type":"ContainerStarted","Data":"47124f879b205ba5c1d879b4b83c9ba1e6a7a7b3ebc3b6f3409a458768fd6205"}
Sep 11 16:35:58.768245 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:35:58.768231    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:35:58.768859 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:35:58.768257    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl" event={"ID":"b5b18639-c0d6-463e-a598-e3d60a378475","Type":"ContainerStarted","Data":"ca7bb59e57a55c5ad9cfa39d7ea827baebbec3df718a5be0a80273105e977792"}
Sep 11 16:35:58.804150 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:35:58.804099    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl" podStartSLOduration=2.610890904 podStartE2EDuration="2m34.804080348s" totalImagesPullingTime="2m2.066097872s" totalInitContainerRuntime="30.127091572s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:35:58.803733778 +0000 UTC m=+1164.143827823" watchObservedRunningTime="2026-09-11 16:35:58.804080348 +0000 UTC m=+1164.144174394"
Sep 11 16:35:59.773478 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:35:59.773431    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:35:59.774769 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:35:59.774741    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl" podUID="b5b18639-c0d6-463e-a598-e3d60a378475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 11 16:36:00.778531 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:00.778486    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl" podUID="b5b18639-c0d6-463e-a598-e3d60a378475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 11 16:36:05.782971 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:05.782938    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:36:05.783586 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:05.783543    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl" podUID="b5b18639-c0d6-463e-a598-e3d60a378475" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 11 16:36:15.784094 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:15.784054    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:36:30.881193 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:30.881144    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"]
Sep 11 16:36:30.881629 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:30.881413    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl" podUID="b5b18639-c0d6-463e-a598-e3d60a378475" containerName="kserve-container" containerID="cri-o://ca7bb59e57a55c5ad9cfa39d7ea827baebbec3df718a5be0a80273105e977792" gracePeriod=30
Sep 11 16:36:30.881629 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:30.881456    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl" podUID="b5b18639-c0d6-463e-a598-e3d60a378475" containerName="kube-rbac-proxy" containerID="cri-o://47124f879b205ba5c1d879b4b83c9ba1e6a7a7b3ebc3b6f3409a458768fd6205" gracePeriod=30
Sep 11 16:36:31.929808 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:31.929771    2561 generic.go:356] "Generic (PLEG): container finished" podID="b5b18639-c0d6-463e-a598-e3d60a378475" containerID="47124f879b205ba5c1d879b4b83c9ba1e6a7a7b3ebc3b6f3409a458768fd6205" exitCode=2
Sep 11 16:36:31.930200 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:31.929845    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl" event={"ID":"b5b18639-c0d6-463e-a598-e3d60a378475","Type":"ContainerDied","Data":"47124f879b205ba5c1d879b4b83c9ba1e6a7a7b3ebc3b6f3409a458768fd6205"}
Sep 11 16:36:32.350131 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.350102    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:36:32.379703 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.379669    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5b18639-c0d6-463e-a598-e3d60a378475-proxy-tls\") pod \"b5b18639-c0d6-463e-a598-e3d60a378475\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") "
Sep 11 16:36:32.379867 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.379711    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5b18639-c0d6-463e-a598-e3d60a378475-cabundle-cert\") pod \"b5b18639-c0d6-463e-a598-e3d60a378475\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") "
Sep 11 16:36:32.379867 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.379760    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5b18639-c0d6-463e-a598-e3d60a378475-kube-rbac-proxy-sar-config\") pod \"b5b18639-c0d6-463e-a598-e3d60a378475\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") "
Sep 11 16:36:32.379867 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.379829    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckszd\" (UniqueName: \"kubernetes.io/projected/b5b18639-c0d6-463e-a598-e3d60a378475-kube-api-access-ckszd\") pod \"b5b18639-c0d6-463e-a598-e3d60a378475\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") "
Sep 11 16:36:32.379867 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.379858    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5b18639-c0d6-463e-a598-e3d60a378475-kserve-provision-location\") pod \"b5b18639-c0d6-463e-a598-e3d60a378475\" (UID: \"b5b18639-c0d6-463e-a598-e3d60a378475\") "
Sep 11 16:36:32.380162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.380135    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b18639-c0d6-463e-a598-e3d60a378475-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5b18639-c0d6-463e-a598-e3d60a378475" (UID: "b5b18639-c0d6-463e-a598-e3d60a378475"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:36:32.380307 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.380213    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b18639-c0d6-463e-a598-e3d60a378475-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b5b18639-c0d6-463e-a598-e3d60a378475" (UID: "b5b18639-c0d6-463e-a598-e3d60a378475"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:36:32.381927 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.381901    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b18639-c0d6-463e-a598-e3d60a378475-kube-api-access-ckszd" (OuterVolumeSpecName: "kube-api-access-ckszd") pod "b5b18639-c0d6-463e-a598-e3d60a378475" (UID: "b5b18639-c0d6-463e-a598-e3d60a378475"). InnerVolumeSpecName "kube-api-access-ckszd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:36:32.382399 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.382367    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b18639-c0d6-463e-a598-e3d60a378475-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5b18639-c0d6-463e-a598-e3d60a378475" (UID: "b5b18639-c0d6-463e-a598-e3d60a378475"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:36:32.384837 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.384809    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b18639-c0d6-463e-a598-e3d60a378475-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5b18639-c0d6-463e-a598-e3d60a378475" (UID: "b5b18639-c0d6-463e-a598-e3d60a378475"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:36:32.480772 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.480674    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckszd\" (UniqueName: \"kubernetes.io/projected/b5b18639-c0d6-463e-a598-e3d60a378475-kube-api-access-ckszd\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:36:32.480772 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.480711    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5b18639-c0d6-463e-a598-e3d60a378475-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:36:32.480772 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.480727    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5b18639-c0d6-463e-a598-e3d60a378475-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:36:32.480772 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.480740    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5b18639-c0d6-463e-a598-e3d60a378475-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:36:32.480772 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.480752    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5b18639-c0d6-463e-a598-e3d60a378475-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:36:32.936896 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.936860    2561 generic.go:356] "Generic (PLEG): container finished" podID="b5b18639-c0d6-463e-a598-e3d60a378475" containerID="ca7bb59e57a55c5ad9cfa39d7ea827baebbec3df718a5be0a80273105e977792" exitCode=0
Sep 11 16:36:32.937367 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.936917    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"
Sep 11 16:36:32.937367 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.936905    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl" event={"ID":"b5b18639-c0d6-463e-a598-e3d60a378475","Type":"ContainerDied","Data":"ca7bb59e57a55c5ad9cfa39d7ea827baebbec3df718a5be0a80273105e977792"}
Sep 11 16:36:32.937367 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.937073    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl" event={"ID":"b5b18639-c0d6-463e-a598-e3d60a378475","Type":"ContainerDied","Data":"55f8cae37fed1d5c9899121f2a42eb8d4be469bd07626e7fc1d423e0dbd4c9a3"}
Sep 11 16:36:32.937367 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.937116    2561 scope.go:117] "RemoveContainer" containerID="47124f879b205ba5c1d879b4b83c9ba1e6a7a7b3ebc3b6f3409a458768fd6205"
Sep 11 16:36:32.946392 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.946377    2561 scope.go:117] "RemoveContainer" containerID="ca7bb59e57a55c5ad9cfa39d7ea827baebbec3df718a5be0a80273105e977792"
Sep 11 16:36:32.954106 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.954085    2561 scope.go:117] "RemoveContainer" containerID="7e005ae6fdec41364cd55095d39a23f0fdbdf91167a0f29376ebcd3bf370d2f6"
Sep 11 16:36:32.962126 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.962107    2561 scope.go:117] "RemoveContainer" containerID="47124f879b205ba5c1d879b4b83c9ba1e6a7a7b3ebc3b6f3409a458768fd6205"
Sep 11 16:36:32.962428 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:36:32.962410    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47124f879b205ba5c1d879b4b83c9ba1e6a7a7b3ebc3b6f3409a458768fd6205\": container with ID starting with 47124f879b205ba5c1d879b4b83c9ba1e6a7a7b3ebc3b6f3409a458768fd6205 not found: ID does not exist" containerID="47124f879b205ba5c1d879b4b83c9ba1e6a7a7b3ebc3b6f3409a458768fd6205"
Sep 11 16:36:32.962499 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.962434    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47124f879b205ba5c1d879b4b83c9ba1e6a7a7b3ebc3b6f3409a458768fd6205"} err="failed to get container status \"47124f879b205ba5c1d879b4b83c9ba1e6a7a7b3ebc3b6f3409a458768fd6205\": rpc error: code = NotFound desc = could not find container \"47124f879b205ba5c1d879b4b83c9ba1e6a7a7b3ebc3b6f3409a458768fd6205\": container with ID starting with 47124f879b205ba5c1d879b4b83c9ba1e6a7a7b3ebc3b6f3409a458768fd6205 not found: ID does not exist"
Sep 11 16:36:32.962499 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.962453    2561 scope.go:117] "RemoveContainer" containerID="ca7bb59e57a55c5ad9cfa39d7ea827baebbec3df718a5be0a80273105e977792"
Sep 11 16:36:32.962702 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:36:32.962685    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca7bb59e57a55c5ad9cfa39d7ea827baebbec3df718a5be0a80273105e977792\": container with ID starting with ca7bb59e57a55c5ad9cfa39d7ea827baebbec3df718a5be0a80273105e977792 not found: ID does not exist" containerID="ca7bb59e57a55c5ad9cfa39d7ea827baebbec3df718a5be0a80273105e977792"
Sep 11 16:36:32.962741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.962706    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7bb59e57a55c5ad9cfa39d7ea827baebbec3df718a5be0a80273105e977792"} err="failed to get container status \"ca7bb59e57a55c5ad9cfa39d7ea827baebbec3df718a5be0a80273105e977792\": rpc error: code = NotFound desc = could not find container \"ca7bb59e57a55c5ad9cfa39d7ea827baebbec3df718a5be0a80273105e977792\": container with ID starting with ca7bb59e57a55c5ad9cfa39d7ea827baebbec3df718a5be0a80273105e977792 not found: ID does not exist"
Sep 11 16:36:32.962741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.962721    2561 scope.go:117] "RemoveContainer" containerID="7e005ae6fdec41364cd55095d39a23f0fdbdf91167a0f29376ebcd3bf370d2f6"
Sep 11 16:36:32.962981 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:36:32.962962    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e005ae6fdec41364cd55095d39a23f0fdbdf91167a0f29376ebcd3bf370d2f6\": container with ID starting with 7e005ae6fdec41364cd55095d39a23f0fdbdf91167a0f29376ebcd3bf370d2f6 not found: ID does not exist" containerID="7e005ae6fdec41364cd55095d39a23f0fdbdf91167a0f29376ebcd3bf370d2f6"
Sep 11 16:36:32.963042 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.962986    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e005ae6fdec41364cd55095d39a23f0fdbdf91167a0f29376ebcd3bf370d2f6"} err="failed to get container status \"7e005ae6fdec41364cd55095d39a23f0fdbdf91167a0f29376ebcd3bf370d2f6\": rpc error: code = NotFound desc = could not find container \"7e005ae6fdec41364cd55095d39a23f0fdbdf91167a0f29376ebcd3bf370d2f6\": container with ID starting with 7e005ae6fdec41364cd55095d39a23f0fdbdf91167a0f29376ebcd3bf370d2f6 not found: ID does not exist"
Sep 11 16:36:32.969057 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.969033    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"]
Sep 11 16:36:32.973712 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:32.973688    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-kvpvl"]
Sep 11 16:36:33.321705 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:33.321615    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5b18639-c0d6-463e-a598-e3d60a378475" path="/var/lib/kubelet/pods/b5b18639-c0d6-463e-a598-e3d60a378475/volumes"
Sep 11 16:36:35.206270 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:35.206244    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:36:35.331772 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:35.331745    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:36:54.440404 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.440361    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"]
Sep 11 16:36:54.440906 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.440891    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b18639-c0d6-463e-a598-e3d60a378475" containerName="storage-initializer"
Sep 11 16:36:54.440991 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.440909    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b18639-c0d6-463e-a598-e3d60a378475" containerName="storage-initializer"
Sep 11 16:36:54.440991 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.440943    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b18639-c0d6-463e-a598-e3d60a378475" containerName="kserve-container"
Sep 11 16:36:54.440991 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.440953    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b18639-c0d6-463e-a598-e3d60a378475" containerName="kserve-container"
Sep 11 16:36:54.440991 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.440965    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b18639-c0d6-463e-a598-e3d60a378475" containerName="kube-rbac-proxy"
Sep 11 16:36:54.440991 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.440974    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b18639-c0d6-463e-a598-e3d60a378475" containerName="kube-rbac-proxy"
Sep 11 16:36:54.441143 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.441066    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b18639-c0d6-463e-a598-e3d60a378475" containerName="kube-rbac-proxy"
Sep 11 16:36:54.441143 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.441077    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b18639-c0d6-463e-a598-e3d60a378475" containerName="kserve-container"
Sep 11 16:36:54.446086 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.446062    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:54.449584 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.449553    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:36:54.449759 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.449636    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 16:36:54.449820 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.449796    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 11 16:36:54.450496 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.450479    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:36:54.451227 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.451198    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:36:54.451314 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.451225    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 11 16:36:54.452021 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.452000    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:36:54.461757 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.461728    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"]
Sep 11 16:36:54.557202 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.557127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8652b6c-e916-4bc8-bd2a-0e073d50b073-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:54.557202 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.557219    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7q4q\" (UniqueName: \"kubernetes.io/projected/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kube-api-access-b7q4q\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:54.557485 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.557246    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8652b6c-e916-4bc8-bd2a-0e073d50b073-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:54.557485 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.557274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:54.557485 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.557364    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:54.658413 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.658371    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8652b6c-e916-4bc8-bd2a-0e073d50b073-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:54.658622 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.658429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7q4q\" (UniqueName: \"kubernetes.io/projected/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kube-api-access-b7q4q\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:54.658622 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.658460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8652b6c-e916-4bc8-bd2a-0e073d50b073-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:54.658622 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.658499    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:54.658622 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.658556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:54.658840 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:36:54.658633    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 11 16:36:54.658840 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:36:54.658702    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8652b6c-e916-4bc8-bd2a-0e073d50b073-proxy-tls podName:e8652b6c-e916-4bc8-bd2a-0e073d50b073 nodeName:}" failed. No retries permitted until 2026-09-11 16:36:55.158680474 +0000 UTC m=+1220.498774505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e8652b6c-e916-4bc8-bd2a-0e073d50b073-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" (UID: "e8652b6c-e916-4bc8-bd2a-0e073d50b073") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 11 16:36:54.658950 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.658899    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:54.659132 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.659111    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8652b6c-e916-4bc8-bd2a-0e073d50b073-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:54.659269 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.659251    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:54.676257 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:54.676216    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7q4q\" (UniqueName: \"kubernetes.io/projected/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kube-api-access-b7q4q\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:55.162697 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:55.162657    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8652b6c-e916-4bc8-bd2a-0e073d50b073-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:55.165183 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:55.165153    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8652b6c-e916-4bc8-bd2a-0e073d50b073-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:55.356581 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:55.356532    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:36:55.639837 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:55.639700    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"]
Sep 11 16:36:55.642592 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:36:55.642555    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8652b6c_e916_4bc8_bd2a_0e073d50b073.slice/crio-2fcc5975731722fbf7e9b1da1162195c242823d9f80dee145a3af7c36ce4bdd3 WatchSource:0}: Error finding container 2fcc5975731722fbf7e9b1da1162195c242823d9f80dee145a3af7c36ce4bdd3: Status 404 returned error can't find the container with id 2fcc5975731722fbf7e9b1da1162195c242823d9f80dee145a3af7c36ce4bdd3
Sep 11 16:36:56.043247 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:56.043208    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" event={"ID":"e8652b6c-e916-4bc8-bd2a-0e073d50b073","Type":"ContainerStarted","Data":"0327fc1164af5c3af6239bb4de86949c00e4d06d5fe71cb0dd840de21c2756b7"}
Sep 11 16:36:56.043451 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:36:56.043272    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" event={"ID":"e8652b6c-e916-4bc8-bd2a-0e073d50b073","Type":"ContainerStarted","Data":"2fcc5975731722fbf7e9b1da1162195c242823d9f80dee145a3af7c36ce4bdd3"}
Sep 11 16:37:00.065190 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:37:00.065079    2561 generic.go:356] "Generic (PLEG): container finished" podID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerID="0327fc1164af5c3af6239bb4de86949c00e4d06d5fe71cb0dd840de21c2756b7" exitCode=0
Sep 11 16:37:00.065190 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:37:00.065140    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" event={"ID":"e8652b6c-e916-4bc8-bd2a-0e073d50b073","Type":"ContainerDied","Data":"0327fc1164af5c3af6239bb4de86949c00e4d06d5fe71cb0dd840de21c2756b7"}
Sep 11 16:37:01.071827 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:37:01.071788    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" event={"ID":"e8652b6c-e916-4bc8-bd2a-0e073d50b073","Type":"ContainerStarted","Data":"c2dea59ffbe14f5178e1799422e542c838b7022c71b5692982c9dbe63a59e166"}
Sep 11 16:37:01.072242 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:37:01.071880    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" event={"ID":"e8652b6c-e916-4bc8-bd2a-0e073d50b073","Type":"ContainerStarted","Data":"43599e67c5dfe887f4ca82bd4c266d0d574103f8217a7c759316a68ef3b11e91"}
Sep 11 16:37:01.072242 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:37:01.072081    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:37:01.072242 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:37:01.072136    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:37:01.096163 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:37:01.096105    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" podStartSLOduration=3.125208377 podStartE2EDuration="7.096088422s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.970880045s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:37:01.094882532 +0000 UTC m=+1226.434976579" watchObservedRunningTime="2026-09-11 16:37:01.096088422 +0000 UTC m=+1226.436182468"
Sep 11 16:37:07.082076 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:37:07.082046    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:37:37.082957 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:37:37.082914    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 11 16:37:47.083462 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:37:47.083413    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 11 16:37:57.082792 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:37:57.082752    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 11 16:38:07.082613 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:07.082568    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 11 16:38:17.149015 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:17.148976    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:38:24.518776 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:24.518738    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"]
Sep 11 16:38:24.519271 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:24.519021    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerName="kserve-container" containerID="cri-o://43599e67c5dfe887f4ca82bd4c266d0d574103f8217a7c759316a68ef3b11e91" gracePeriod=30
Sep 11 16:38:24.519271 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:24.519037    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerName="kube-rbac-proxy" containerID="cri-o://c2dea59ffbe14f5178e1799422e542c838b7022c71b5692982c9dbe63a59e166" gracePeriod=30
Sep 11 16:38:25.472999 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:25.472964    2561 generic.go:356] "Generic (PLEG): container finished" podID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerID="c2dea59ffbe14f5178e1799422e542c838b7022c71b5692982c9dbe63a59e166" exitCode=2
Sep 11 16:38:25.473196 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:25.473010    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" event={"ID":"e8652b6c-e916-4bc8-bd2a-0e073d50b073","Type":"ContainerDied","Data":"c2dea59ffbe14f5178e1799422e542c838b7022c71b5692982c9dbe63a59e166"}
Sep 11 16:38:27.077479 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:27.077438    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 11 16:38:27.082880 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:27.082848    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 11 16:38:29.494291 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:29.494254    2561 generic.go:356] "Generic (PLEG): container finished" podID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerID="43599e67c5dfe887f4ca82bd4c266d0d574103f8217a7c759316a68ef3b11e91" exitCode=0
Sep 11 16:38:29.494651 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:29.494326    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" event={"ID":"e8652b6c-e916-4bc8-bd2a-0e073d50b073","Type":"ContainerDied","Data":"43599e67c5dfe887f4ca82bd4c266d0d574103f8217a7c759316a68ef3b11e91"}
Sep 11 16:38:29.776567 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:29.776541    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:38:29.910314 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:29.910279    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8652b6c-e916-4bc8-bd2a-0e073d50b073-cabundle-cert\") pod \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") "
Sep 11 16:38:29.910314 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:29.910329    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kube-rbac-proxy-sar-config\") pod \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") "
Sep 11 16:38:29.910700 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:29.910386    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kserve-provision-location\") pod \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") "
Sep 11 16:38:29.910700 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:29.910410    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7q4q\" (UniqueName: \"kubernetes.io/projected/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kube-api-access-b7q4q\") pod \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") "
Sep 11 16:38:29.910700 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:29.910471    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8652b6c-e916-4bc8-bd2a-0e073d50b073-proxy-tls\") pod \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\" (UID: \"e8652b6c-e916-4bc8-bd2a-0e073d50b073\") "
Sep 11 16:38:29.910859 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:29.910729    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e8652b6c-e916-4bc8-bd2a-0e073d50b073" (UID: "e8652b6c-e916-4bc8-bd2a-0e073d50b073"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:38:29.910859 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:29.910770    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e8652b6c-e916-4bc8-bd2a-0e073d50b073" (UID: "e8652b6c-e916-4bc8-bd2a-0e073d50b073"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:38:29.910859 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:29.910779    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8652b6c-e916-4bc8-bd2a-0e073d50b073-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e8652b6c-e916-4bc8-bd2a-0e073d50b073" (UID: "e8652b6c-e916-4bc8-bd2a-0e073d50b073"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:38:29.912567 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:29.912537    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8652b6c-e916-4bc8-bd2a-0e073d50b073-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8652b6c-e916-4bc8-bd2a-0e073d50b073" (UID: "e8652b6c-e916-4bc8-bd2a-0e073d50b073"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:38:29.912703 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:29.912682    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kube-api-access-b7q4q" (OuterVolumeSpecName: "kube-api-access-b7q4q") pod "e8652b6c-e916-4bc8-bd2a-0e073d50b073" (UID: "e8652b6c-e916-4bc8-bd2a-0e073d50b073"). InnerVolumeSpecName "kube-api-access-b7q4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:38:30.011693 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:30.011592    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:38:30.011693 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:30.011628    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7q4q\" (UniqueName: \"kubernetes.io/projected/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kube-api-access-b7q4q\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:38:30.011693 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:30.011643    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8652b6c-e916-4bc8-bd2a-0e073d50b073-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:38:30.011693 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:30.011656    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8652b6c-e916-4bc8-bd2a-0e073d50b073-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:38:30.011693 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:30.011667    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8652b6c-e916-4bc8-bd2a-0e073d50b073-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:38:30.501397 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:30.501354    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2" event={"ID":"e8652b6c-e916-4bc8-bd2a-0e073d50b073","Type":"ContainerDied","Data":"2fcc5975731722fbf7e9b1da1162195c242823d9f80dee145a3af7c36ce4bdd3"}
Sep 11 16:38:30.501828 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:30.501379    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"
Sep 11 16:38:30.501828 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:30.501432    2561 scope.go:117] "RemoveContainer" containerID="c2dea59ffbe14f5178e1799422e542c838b7022c71b5692982c9dbe63a59e166"
Sep 11 16:38:30.510249 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:30.510225    2561 scope.go:117] "RemoveContainer" containerID="43599e67c5dfe887f4ca82bd4c266d0d574103f8217a7c759316a68ef3b11e91"
Sep 11 16:38:30.518275 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:30.518255    2561 scope.go:117] "RemoveContainer" containerID="0327fc1164af5c3af6239bb4de86949c00e4d06d5fe71cb0dd840de21c2756b7"
Sep 11 16:38:30.525602 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:30.525577    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"]
Sep 11 16:38:30.530633 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:30.530610    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-tnzj2"]
Sep 11 16:38:31.321494 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:31.321460    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" path="/var/lib/kubelet/pods/e8652b6c-e916-4bc8-bd2a-0e073d50b073/volumes"
Sep 11 16:38:47.269271 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.269233    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-ae539cb5"]
Sep 11 16:38:47.269692 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.269573    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerName="kserve-container"
Sep 11 16:38:47.269692 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.269583    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerName="kserve-container"
Sep 11 16:38:47.269692 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.269601    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerName="storage-initializer"
Sep 11 16:38:47.269692 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.269607    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerName="storage-initializer"
Sep 11 16:38:47.269692 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.269613    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerName="kube-rbac-proxy"
Sep 11 16:38:47.269692 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.269619    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerName="kube-rbac-proxy"
Sep 11 16:38:47.269692 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.269688    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerName="kserve-container"
Sep 11 16:38:47.269692 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.269696    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8652b6c-e916-4bc8-bd2a-0e073d50b073" containerName="kube-rbac-proxy"
Sep 11 16:38:47.273756 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.273740    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-ae539cb5"
Sep 11 16:38:47.276091 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.276063    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 11 16:38:47.276225 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.276136    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 11 16:38:47.276651 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.276637    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q9d2b\""
Sep 11 16:38:47.287885 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.287853    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-ae539cb5"]
Sep 11 16:38:47.367337 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.367297    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mngb5\" (UniqueName: \"kubernetes.io/projected/1c2e059c-24c6-4875-9745-de4988eb017d-kube-api-access-mngb5\") pod \"curl-metrics-ae539cb5\" (UID: \"1c2e059c-24c6-4875-9745-de4988eb017d\") " pod="kserve/curl-metrics-ae539cb5"
Sep 11 16:38:47.468186 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.468137    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mngb5\" (UniqueName: \"kubernetes.io/projected/1c2e059c-24c6-4875-9745-de4988eb017d-kube-api-access-mngb5\") pod \"curl-metrics-ae539cb5\" (UID: \"1c2e059c-24c6-4875-9745-de4988eb017d\") " pod="kserve/curl-metrics-ae539cb5"
Sep 11 16:38:47.484683 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.484648    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mngb5\" (UniqueName: \"kubernetes.io/projected/1c2e059c-24c6-4875-9745-de4988eb017d-kube-api-access-mngb5\") pod \"curl-metrics-ae539cb5\" (UID: \"1c2e059c-24c6-4875-9745-de4988eb017d\") " pod="kserve/curl-metrics-ae539cb5"
Sep 11 16:38:47.597075 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.596987    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-ae539cb5"
Sep 11 16:38:47.875642 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:47.875556    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-ae539cb5"]
Sep 11 16:38:47.878812 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:38:47.878776    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c2e059c_24c6_4875_9745_de4988eb017d.slice/crio-90fb9431117856196f373c773a3834a1ae0bc838060a1a8af4d50f98ad6f89d9 WatchSource:0}: Error finding container 90fb9431117856196f373c773a3834a1ae0bc838060a1a8af4d50f98ad6f89d9: Status 404 returned error can't find the container with id 90fb9431117856196f373c773a3834a1ae0bc838060a1a8af4d50f98ad6f89d9
Sep 11 16:38:48.590240 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:48.590210    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-ae539cb5" event={"ID":"1c2e059c-24c6-4875-9745-de4988eb017d","Type":"ContainerStarted","Data":"90fb9431117856196f373c773a3834a1ae0bc838060a1a8af4d50f98ad6f89d9"}
Sep 11 16:38:49.595870 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:49.595829    2561 generic.go:356] "Generic (PLEG): container finished" podID="1c2e059c-24c6-4875-9745-de4988eb017d" containerID="4d48e94a4dd4575d08ee7f32e13f4a186ddcfc72de1c7b6dfadf431495c6fd00" exitCode=0
Sep 11 16:38:49.596272 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:49.595918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-ae539cb5" event={"ID":"1c2e059c-24c6-4875-9745-de4988eb017d","Type":"ContainerDied","Data":"4d48e94a4dd4575d08ee7f32e13f4a186ddcfc72de1c7b6dfadf431495c6fd00"}
Sep 11 16:38:50.729331 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:50.729304    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-ae539cb5"
Sep 11 16:38:50.777276 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:50.777244    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-ae539cb5_1c2e059c-24c6-4875-9745-de4988eb017d/curl-metrics-ae539cb5/0.log"
Sep 11 16:38:50.815412 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:50.815377    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-ae539cb5"]
Sep 11 16:38:50.821467 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:50.821421    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-ae539cb5"]
Sep 11 16:38:50.897452 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:50.897360    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mngb5\" (UniqueName: \"kubernetes.io/projected/1c2e059c-24c6-4875-9745-de4988eb017d-kube-api-access-mngb5\") pod \"1c2e059c-24c6-4875-9745-de4988eb017d\" (UID: \"1c2e059c-24c6-4875-9745-de4988eb017d\") "
Sep 11 16:38:50.899433 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:50.899398    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2e059c-24c6-4875-9745-de4988eb017d-kube-api-access-mngb5" (OuterVolumeSpecName: "kube-api-access-mngb5") pod "1c2e059c-24c6-4875-9745-de4988eb017d" (UID: "1c2e059c-24c6-4875-9745-de4988eb017d"). InnerVolumeSpecName "kube-api-access-mngb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:38:50.998667 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:50.998628    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mngb5\" (UniqueName: \"kubernetes.io/projected/1c2e059c-24c6-4875-9745-de4988eb017d-kube-api-access-mngb5\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:38:51.321555 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:51.321520    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c2e059c-24c6-4875-9745-de4988eb017d" path="/var/lib/kubelet/pods/1c2e059c-24c6-4875-9745-de4988eb017d/volumes"
Sep 11 16:38:51.606014 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:51.605937    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-ae539cb5"
Sep 11 16:38:51.606014 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:51.605975    2561 scope.go:117] "RemoveContainer" containerID="4d48e94a4dd4575d08ee7f32e13f4a186ddcfc72de1c7b6dfadf431495c6fd00"
Sep 11 16:38:56.214982 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:56.214948    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-6cd33963"]
Sep 11 16:38:56.215366 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:56.215306    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2e059c-24c6-4875-9745-de4988eb017d" containerName="curl-metrics-ae539cb5"
Sep 11 16:38:56.215366 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:56.215318    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2e059c-24c6-4875-9745-de4988eb017d" containerName="curl-metrics-ae539cb5"
Sep 11 16:38:56.215450 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:56.215379    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2e059c-24c6-4875-9745-de4988eb017d" containerName="curl-metrics-ae539cb5"
Sep 11 16:38:56.219658 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:56.219642    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-6cd33963"
Sep 11 16:38:56.222141 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:56.222120    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 11 16:38:56.222716 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:56.222699    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q9d2b\""
Sep 11 16:38:56.222716 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:56.222713    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 11 16:38:56.225766 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:56.225720    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-6cd33963"]
Sep 11 16:38:56.347069 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:56.347031    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq4c9\" (UniqueName: \"kubernetes.io/projected/b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f-kube-api-access-gq4c9\") pod \"curl-metrics-6cd33963\" (UID: \"b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f\") " pod="kserve/curl-metrics-6cd33963"
Sep 11 16:38:56.447543 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:56.447506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4c9\" (UniqueName: \"kubernetes.io/projected/b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f-kube-api-access-gq4c9\") pod \"curl-metrics-6cd33963\" (UID: \"b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f\") " pod="kserve/curl-metrics-6cd33963"
Sep 11 16:38:56.459092 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:56.459064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq4c9\" (UniqueName: \"kubernetes.io/projected/b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f-kube-api-access-gq4c9\") pod \"curl-metrics-6cd33963\" (UID: \"b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f\") " pod="kserve/curl-metrics-6cd33963"
Sep 11 16:38:56.538711 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:56.538632    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-6cd33963"
Sep 11 16:38:56.812739 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:56.812661    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-6cd33963"]
Sep 11 16:38:56.816051 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:38:56.816021    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1375ee4_b6c3_4cec_a7eb_2bdc48a4862f.slice/crio-c64bf36be950442c499af0f943da006c77cd79780b0bf8ed0b92f5b54ac10170 WatchSource:0}: Error finding container c64bf36be950442c499af0f943da006c77cd79780b0bf8ed0b92f5b54ac10170: Status 404 returned error can't find the container with id c64bf36be950442c499af0f943da006c77cd79780b0bf8ed0b92f5b54ac10170
Sep 11 16:38:57.634819 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:57.634784    2561 generic.go:356] "Generic (PLEG): container finished" podID="b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f" containerID="b4f6ae138df18f9cfcf0789064c879f5c85764325e0695542d4c52628059ccbc" exitCode=0
Sep 11 16:38:57.635273 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:57.634855    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-6cd33963" event={"ID":"b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f","Type":"ContainerDied","Data":"b4f6ae138df18f9cfcf0789064c879f5c85764325e0695542d4c52628059ccbc"}
Sep 11 16:38:57.635273 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:57.634883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-6cd33963" event={"ID":"b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f","Type":"ContainerStarted","Data":"c64bf36be950442c499af0f943da006c77cd79780b0bf8ed0b92f5b54ac10170"}
Sep 11 16:38:58.772705 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:58.772674    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-6cd33963"
Sep 11 16:38:58.802235 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:58.802203    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-6cd33963_b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f/curl-metrics-6cd33963/0.log"
Sep 11 16:38:58.829978 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:58.829936    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-6cd33963"]
Sep 11 16:38:58.835597 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:58.835574    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-6cd33963"]
Sep 11 16:38:58.869971 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:58.869945    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq4c9\" (UniqueName: \"kubernetes.io/projected/b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f-kube-api-access-gq4c9\") pod \"b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f\" (UID: \"b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f\") "
Sep 11 16:38:58.872024 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:58.871997    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f-kube-api-access-gq4c9" (OuterVolumeSpecName: "kube-api-access-gq4c9") pod "b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f" (UID: "b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f"). InnerVolumeSpecName "kube-api-access-gq4c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:38:58.970532 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:58.970488    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq4c9\" (UniqueName: \"kubernetes.io/projected/b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f-kube-api-access-gq4c9\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:38:59.320927 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:59.320849    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f" path="/var/lib/kubelet/pods/b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f/volumes"
Sep 11 16:38:59.645146 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:59.645067    2561 scope.go:117] "RemoveContainer" containerID="b4f6ae138df18f9cfcf0789064c879f5c85764325e0695542d4c52628059ccbc"
Sep 11 16:38:59.645146 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:38:59.645076    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-6cd33963"
Sep 11 16:40:49.564277 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.564239    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"]
Sep 11 16:40:49.564754 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.564660    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f" containerName="curl-metrics-6cd33963"
Sep 11 16:40:49.564754 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.564675    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f" containerName="curl-metrics-6cd33963"
Sep 11 16:40:49.564754 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.564742    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1375ee4-b6c3-4cec-a7eb-2bdc48a4862f" containerName="curl-metrics-6cd33963"
Sep 11 16:40:49.567928 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.567908    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:49.570458 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.570423    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 11 16:40:49.570598 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.570503    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:40:49.570598 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.570539    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:40:49.570699 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.570613    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 16:40:49.570870 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.570852    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 11 16:40:49.571151 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.571136    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:40:49.571393 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.571221    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:40:49.582694 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.582664    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"]
Sep 11 16:40:49.620697 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.620660    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bslh9\" (UniqueName: \"kubernetes.io/projected/8a127288-cc02-4058-b9df-7629dcbce64c-kube-api-access-bslh9\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-44txf\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:49.620894 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.620714    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a127288-cc02-4058-b9df-7629dcbce64c-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-44txf\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:49.620894 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.620769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a127288-cc02-4058-b9df-7629dcbce64c-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-44txf\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:49.620894 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.620825    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a127288-cc02-4058-b9df-7629dcbce64c-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-44txf\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:49.620894 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.620858    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a127288-cc02-4058-b9df-7629dcbce64c-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-44txf\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:49.721275 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.721237    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a127288-cc02-4058-b9df-7629dcbce64c-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-44txf\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:49.721466 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.721289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a127288-cc02-4058-b9df-7629dcbce64c-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-44txf\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:49.721466 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.721315    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bslh9\" (UniqueName: \"kubernetes.io/projected/8a127288-cc02-4058-b9df-7629dcbce64c-kube-api-access-bslh9\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-44txf\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:49.721466 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.721349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a127288-cc02-4058-b9df-7629dcbce64c-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-44txf\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:49.721466 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.721370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a127288-cc02-4058-b9df-7629dcbce64c-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-44txf\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:49.721682 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:40:49.721484    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 11 16:40:49.721682 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:40:49.721570    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a127288-cc02-4058-b9df-7629dcbce64c-proxy-tls podName:8a127288-cc02-4058-b9df-7629dcbce64c nodeName:}" failed. No retries permitted until 2026-09-11 16:40:50.221544579 +0000 UTC m=+1455.561638611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8a127288-cc02-4058-b9df-7629dcbce64c-proxy-tls") pod "sklearn-v2-mlserver-predictor-7f568cdc6b-44txf" (UID: "8a127288-cc02-4058-b9df-7629dcbce64c") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 11 16:40:49.721790 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.721703    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a127288-cc02-4058-b9df-7629dcbce64c-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-44txf\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:49.722014 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.721993    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a127288-cc02-4058-b9df-7629dcbce64c-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-44txf\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:49.722063 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.722043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a127288-cc02-4058-b9df-7629dcbce64c-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-44txf\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:49.730515 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:49.730490    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bslh9\" (UniqueName: \"kubernetes.io/projected/8a127288-cc02-4058-b9df-7629dcbce64c-kube-api-access-bslh9\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-44txf\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:50.226117 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:50.226065    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a127288-cc02-4058-b9df-7629dcbce64c-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-44txf\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:50.228531 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:50.228492    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a127288-cc02-4058-b9df-7629dcbce64c-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-44txf\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:50.486507 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:50.486389    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:40:50.766236 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:50.766169    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"]
Sep 11 16:40:50.769032 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:40:50.769003    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a127288_cc02_4058_b9df_7629dcbce64c.slice/crio-3709d35d2209114a27d2f8b0250908f681a727acfe5c260d1f20e879ff5371b7 WatchSource:0}: Error finding container 3709d35d2209114a27d2f8b0250908f681a727acfe5c260d1f20e879ff5371b7: Status 404 returned error can't find the container with id 3709d35d2209114a27d2f8b0250908f681a727acfe5c260d1f20e879ff5371b7
Sep 11 16:40:51.139624 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:51.139525    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf" event={"ID":"8a127288-cc02-4058-b9df-7629dcbce64c","Type":"ContainerStarted","Data":"ef7ea0cb49673206f297cfc91509fe4f5f25e667ed4deffe42f9c130152a942e"}
Sep 11 16:40:51.139624 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:51.139589    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf" event={"ID":"8a127288-cc02-4058-b9df-7629dcbce64c","Type":"ContainerStarted","Data":"3709d35d2209114a27d2f8b0250908f681a727acfe5c260d1f20e879ff5371b7"}
Sep 11 16:40:55.159144 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:55.159113    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a127288-cc02-4058-b9df-7629dcbce64c" containerID="ef7ea0cb49673206f297cfc91509fe4f5f25e667ed4deffe42f9c130152a942e" exitCode=0
Sep 11 16:40:55.159612 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:55.159163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf" event={"ID":"8a127288-cc02-4058-b9df-7629dcbce64c","Type":"ContainerDied","Data":"ef7ea0cb49673206f297cfc91509fe4f5f25e667ed4deffe42f9c130152a942e"}
Sep 11 16:40:55.160267 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:40:55.160245    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:41:35.244893 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:41:35.244867    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:41:35.368609 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:41:35.368583    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:43:03.886787 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:43:03.886748    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf" event={"ID":"8a127288-cc02-4058-b9df-7629dcbce64c","Type":"ContainerStarted","Data":"2e351bfe2f67bdbfefcce4bedf3efbdd64f0458fb407a687025982abeded0e77"}
Sep 11 16:43:03.887281 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:43:03.886808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf" event={"ID":"8a127288-cc02-4058-b9df-7629dcbce64c","Type":"ContainerStarted","Data":"9206139407a61fc31077c55a8d3162b1a87d676ef599401e55c0d394fa49516e"}
Sep 11 16:43:03.887281 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:43:03.886926    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:43:03.909926 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:43:03.909876    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf" podStartSLOduration=3.166031236 podStartE2EDuration="2m14.909860449s" totalImagesPullingTime="2m7.986363326s" totalInitContainerRuntime="3.757465887s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:03.908235367 +0000 UTC m=+1589.248329414" watchObservedRunningTime="2026-09-11 16:43:03.909860449 +0000 UTC m=+1589.249954565"
Sep 11 16:43:04.892703 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:43:04.892618    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:43:10.902905 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:43:10.902872    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:43:40.942286 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:43:40.942245    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf" podUID="8a127288-cc02-4058-b9df-7629dcbce64c" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 11 16:43:50.905346 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:43:50.905305    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:43:59.825101 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:43:59.825067    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"]
Sep 11 16:43:59.825506 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:43:59.825319    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf" podUID="8a127288-cc02-4058-b9df-7629dcbce64c" containerName="kserve-container" containerID="cri-o://9206139407a61fc31077c55a8d3162b1a87d676ef599401e55c0d394fa49516e" gracePeriod=30
Sep 11 16:43:59.825506 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:43:59.825358    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf" podUID="8a127288-cc02-4058-b9df-7629dcbce64c" containerName="kube-rbac-proxy" containerID="cri-o://2e351bfe2f67bdbfefcce4bedf3efbdd64f0458fb407a687025982abeded0e77" gracePeriod=30
Sep 11 16:44:00.164098 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:00.164068    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a127288-cc02-4058-b9df-7629dcbce64c" containerID="2e351bfe2f67bdbfefcce4bedf3efbdd64f0458fb407a687025982abeded0e77" exitCode=2
Sep 11 16:44:00.164285 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:00.164136    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf" event={"ID":"8a127288-cc02-4058-b9df-7629dcbce64c","Type":"ContainerDied","Data":"2e351bfe2f67bdbfefcce4bedf3efbdd64f0458fb407a687025982abeded0e77"}
Sep 11 16:44:00.899153 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:00.899110    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf" podUID="8a127288-cc02-4058-b9df-7629dcbce64c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 11 16:44:01.945393 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:01.945345    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf" podUID="8a127288-cc02-4058-b9df-7629dcbce64c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.35:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 16:44:05.898408 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:05.898364    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf" podUID="8a127288-cc02-4058-b9df-7629dcbce64c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 11 16:44:07.173336 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.173310    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:44:07.200322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.200241    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a127288-cc02-4058-b9df-7629dcbce64c" containerID="9206139407a61fc31077c55a8d3162b1a87d676ef599401e55c0d394fa49516e" exitCode=0
Sep 11 16:44:07.200322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.200298    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"
Sep 11 16:44:07.200508 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.200315    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf" event={"ID":"8a127288-cc02-4058-b9df-7629dcbce64c","Type":"ContainerDied","Data":"9206139407a61fc31077c55a8d3162b1a87d676ef599401e55c0d394fa49516e"}
Sep 11 16:44:07.200508 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.200394    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf" event={"ID":"8a127288-cc02-4058-b9df-7629dcbce64c","Type":"ContainerDied","Data":"3709d35d2209114a27d2f8b0250908f681a727acfe5c260d1f20e879ff5371b7"}
Sep 11 16:44:07.200508 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.200432    2561 scope.go:117] "RemoveContainer" containerID="2e351bfe2f67bdbfefcce4bedf3efbdd64f0458fb407a687025982abeded0e77"
Sep 11 16:44:07.208718 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.208695    2561 scope.go:117] "RemoveContainer" containerID="9206139407a61fc31077c55a8d3162b1a87d676ef599401e55c0d394fa49516e"
Sep 11 16:44:07.217199 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.217152    2561 scope.go:117] "RemoveContainer" containerID="ef7ea0cb49673206f297cfc91509fe4f5f25e667ed4deffe42f9c130152a942e"
Sep 11 16:44:07.224782 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.224764    2561 scope.go:117] "RemoveContainer" containerID="2e351bfe2f67bdbfefcce4bedf3efbdd64f0458fb407a687025982abeded0e77"
Sep 11 16:44:07.225067 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:44:07.225047    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e351bfe2f67bdbfefcce4bedf3efbdd64f0458fb407a687025982abeded0e77\": container with ID starting with 2e351bfe2f67bdbfefcce4bedf3efbdd64f0458fb407a687025982abeded0e77 not found: ID does not exist" containerID="2e351bfe2f67bdbfefcce4bedf3efbdd64f0458fb407a687025982abeded0e77"
Sep 11 16:44:07.225131 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.225077    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e351bfe2f67bdbfefcce4bedf3efbdd64f0458fb407a687025982abeded0e77"} err="failed to get container status \"2e351bfe2f67bdbfefcce4bedf3efbdd64f0458fb407a687025982abeded0e77\": rpc error: code = NotFound desc = could not find container \"2e351bfe2f67bdbfefcce4bedf3efbdd64f0458fb407a687025982abeded0e77\": container with ID starting with 2e351bfe2f67bdbfefcce4bedf3efbdd64f0458fb407a687025982abeded0e77 not found: ID does not exist"
Sep 11 16:44:07.225131 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.225095    2561 scope.go:117] "RemoveContainer" containerID="9206139407a61fc31077c55a8d3162b1a87d676ef599401e55c0d394fa49516e"
Sep 11 16:44:07.225394 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:44:07.225377    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9206139407a61fc31077c55a8d3162b1a87d676ef599401e55c0d394fa49516e\": container with ID starting with 9206139407a61fc31077c55a8d3162b1a87d676ef599401e55c0d394fa49516e not found: ID does not exist" containerID="9206139407a61fc31077c55a8d3162b1a87d676ef599401e55c0d394fa49516e"
Sep 11 16:44:07.225439 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.225400    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9206139407a61fc31077c55a8d3162b1a87d676ef599401e55c0d394fa49516e"} err="failed to get container status \"9206139407a61fc31077c55a8d3162b1a87d676ef599401e55c0d394fa49516e\": rpc error: code = NotFound desc = could not find container \"9206139407a61fc31077c55a8d3162b1a87d676ef599401e55c0d394fa49516e\": container with ID starting with 9206139407a61fc31077c55a8d3162b1a87d676ef599401e55c0d394fa49516e not found: ID does not exist"
Sep 11 16:44:07.225439 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.225415    2561 scope.go:117] "RemoveContainer" containerID="ef7ea0cb49673206f297cfc91509fe4f5f25e667ed4deffe42f9c130152a942e"
Sep 11 16:44:07.225658 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:44:07.225643    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef7ea0cb49673206f297cfc91509fe4f5f25e667ed4deffe42f9c130152a942e\": container with ID starting with ef7ea0cb49673206f297cfc91509fe4f5f25e667ed4deffe42f9c130152a942e not found: ID does not exist" containerID="ef7ea0cb49673206f297cfc91509fe4f5f25e667ed4deffe42f9c130152a942e"
Sep 11 16:44:07.225723 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.225662    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef7ea0cb49673206f297cfc91509fe4f5f25e667ed4deffe42f9c130152a942e"} err="failed to get container status \"ef7ea0cb49673206f297cfc91509fe4f5f25e667ed4deffe42f9c130152a942e\": rpc error: code = NotFound desc = could not find container \"ef7ea0cb49673206f297cfc91509fe4f5f25e667ed4deffe42f9c130152a942e\": container with ID starting with ef7ea0cb49673206f297cfc91509fe4f5f25e667ed4deffe42f9c130152a942e not found: ID does not exist"
Sep 11 16:44:07.243481 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.243459    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a127288-cc02-4058-b9df-7629dcbce64c-kube-rbac-proxy-sar-config\") pod \"8a127288-cc02-4058-b9df-7629dcbce64c\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") "
Sep 11 16:44:07.243548 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.243486    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a127288-cc02-4058-b9df-7629dcbce64c-cabundle-cert\") pod \"8a127288-cc02-4058-b9df-7629dcbce64c\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") "
Sep 11 16:44:07.243548 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.243517    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a127288-cc02-4058-b9df-7629dcbce64c-proxy-tls\") pod \"8a127288-cc02-4058-b9df-7629dcbce64c\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") "
Sep 11 16:44:07.243621 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.243558    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a127288-cc02-4058-b9df-7629dcbce64c-kserve-provision-location\") pod \"8a127288-cc02-4058-b9df-7629dcbce64c\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") "
Sep 11 16:44:07.243621 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.243589    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bslh9\" (UniqueName: \"kubernetes.io/projected/8a127288-cc02-4058-b9df-7629dcbce64c-kube-api-access-bslh9\") pod \"8a127288-cc02-4058-b9df-7629dcbce64c\" (UID: \"8a127288-cc02-4058-b9df-7629dcbce64c\") "
Sep 11 16:44:07.243875 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.243848    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a127288-cc02-4058-b9df-7629dcbce64c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a127288-cc02-4058-b9df-7629dcbce64c" (UID: "8a127288-cc02-4058-b9df-7629dcbce64c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:44:07.243945 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.243881    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a127288-cc02-4058-b9df-7629dcbce64c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8a127288-cc02-4058-b9df-7629dcbce64c" (UID: "8a127288-cc02-4058-b9df-7629dcbce64c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:44:07.243945 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.243899    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a127288-cc02-4058-b9df-7629dcbce64c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a127288-cc02-4058-b9df-7629dcbce64c" (UID: "8a127288-cc02-4058-b9df-7629dcbce64c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:44:07.245670 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.245646    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a127288-cc02-4058-b9df-7629dcbce64c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a127288-cc02-4058-b9df-7629dcbce64c" (UID: "8a127288-cc02-4058-b9df-7629dcbce64c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:44:07.245757 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.245645    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a127288-cc02-4058-b9df-7629dcbce64c-kube-api-access-bslh9" (OuterVolumeSpecName: "kube-api-access-bslh9") pod "8a127288-cc02-4058-b9df-7629dcbce64c" (UID: "8a127288-cc02-4058-b9df-7629dcbce64c"). InnerVolumeSpecName "kube-api-access-bslh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:44:07.345246 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.345217    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a127288-cc02-4058-b9df-7629dcbce64c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:44:07.345246 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.345245    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a127288-cc02-4058-b9df-7629dcbce64c-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:44:07.345394 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.345258    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a127288-cc02-4058-b9df-7629dcbce64c-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:44:07.345394 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.345267    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a127288-cc02-4058-b9df-7629dcbce64c-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:44:07.345394 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.345275    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bslh9\" (UniqueName: \"kubernetes.io/projected/8a127288-cc02-4058-b9df-7629dcbce64c-kube-api-access-bslh9\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:44:07.519906 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.519813    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"]
Sep 11 16:44:07.522427 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:07.522400    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-44txf"]
Sep 11 16:44:09.320958 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:09.320924    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a127288-cc02-4058-b9df-7629dcbce64c" path="/var/lib/kubelet/pods/8a127288-cc02-4058-b9df-7629dcbce64c/volumes"
Sep 11 16:44:42.089703 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.089663    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"]
Sep 11 16:44:42.090198 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.090013    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a127288-cc02-4058-b9df-7629dcbce64c" containerName="kserve-container"
Sep 11 16:44:42.090198 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.090024    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a127288-cc02-4058-b9df-7629dcbce64c" containerName="kserve-container"
Sep 11 16:44:42.090198 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.090032    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a127288-cc02-4058-b9df-7629dcbce64c" containerName="storage-initializer"
Sep 11 16:44:42.090198 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.090037    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a127288-cc02-4058-b9df-7629dcbce64c" containerName="storage-initializer"
Sep 11 16:44:42.090198 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.090055    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a127288-cc02-4058-b9df-7629dcbce64c" containerName="kube-rbac-proxy"
Sep 11 16:44:42.090198 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.090060    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a127288-cc02-4058-b9df-7629dcbce64c" containerName="kube-rbac-proxy"
Sep 11 16:44:42.090198 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.090113    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a127288-cc02-4058-b9df-7629dcbce64c" containerName="kserve-container"
Sep 11 16:44:42.090198 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.090123    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a127288-cc02-4058-b9df-7629dcbce64c" containerName="kube-rbac-proxy"
Sep 11 16:44:42.094512 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.094491    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.096999 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.096972    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 11 16:44:42.097131 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.096972    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:44:42.097131 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.097019    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:44:42.097131 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.096976    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 11 16:44:42.097282 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.097166    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:44:42.097574 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.097555    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 16:44:42.097645 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.097562    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:44:42.105080 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.105058    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"]
Sep 11 16:44:42.144048 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.144018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed94223d-c737-452e-a64c-a7ea1a2248aa-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.144243 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.144066    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed94223d-c737-452e-a64c-a7ea1a2248aa-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.144243 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.144194    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px9kf\" (UniqueName: \"kubernetes.io/projected/ed94223d-c737-452e-a64c-a7ea1a2248aa-kube-api-access-px9kf\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.144348 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.144262    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed94223d-c737-452e-a64c-a7ea1a2248aa-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.144348 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.144288    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed94223d-c737-452e-a64c-a7ea1a2248aa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.245032 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.244950    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9kf\" (UniqueName: \"kubernetes.io/projected/ed94223d-c737-452e-a64c-a7ea1a2248aa-kube-api-access-px9kf\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.245032 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.245005    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed94223d-c737-452e-a64c-a7ea1a2248aa-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.245032 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.245029    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed94223d-c737-452e-a64c-a7ea1a2248aa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.245297 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.245059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed94223d-c737-452e-a64c-a7ea1a2248aa-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.245297 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.245092    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed94223d-c737-452e-a64c-a7ea1a2248aa-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.245601 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.245580    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed94223d-c737-452e-a64c-a7ea1a2248aa-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.245901 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.245883    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed94223d-c737-452e-a64c-a7ea1a2248aa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.245957 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.245895    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed94223d-c737-452e-a64c-a7ea1a2248aa-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.247651 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.247633    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed94223d-c737-452e-a64c-a7ea1a2248aa-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.255895 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.255869    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9kf\" (UniqueName: \"kubernetes.io/projected/ed94223d-c737-452e-a64c-a7ea1a2248aa-kube-api-access-px9kf\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.406975 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.406939    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:42.682214 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:42.682163    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"]
Sep 11 16:44:42.685111 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:44:42.685085    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded94223d_c737_452e_a64c_a7ea1a2248aa.slice/crio-097f5172dc2a8ba42654ebe24b80c5aa5133a4888451035914456dcfe45a29bc WatchSource:0}: Error finding container 097f5172dc2a8ba42654ebe24b80c5aa5133a4888451035914456dcfe45a29bc: Status 404 returned error can't find the container with id 097f5172dc2a8ba42654ebe24b80c5aa5133a4888451035914456dcfe45a29bc
Sep 11 16:44:43.368567 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:43.368531    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9" event={"ID":"ed94223d-c737-452e-a64c-a7ea1a2248aa","Type":"ContainerStarted","Data":"bedf610b7cfb08ee1ac40807823ae9cc3bc8ccc7b1d1b252de32076d4f20d6e1"}
Sep 11 16:44:43.368971 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:43.368586    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9" event={"ID":"ed94223d-c737-452e-a64c-a7ea1a2248aa","Type":"ContainerStarted","Data":"097f5172dc2a8ba42654ebe24b80c5aa5133a4888451035914456dcfe45a29bc"}
Sep 11 16:44:47.390080 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:47.390031    2561 generic.go:356] "Generic (PLEG): container finished" podID="ed94223d-c737-452e-a64c-a7ea1a2248aa" containerID="bedf610b7cfb08ee1ac40807823ae9cc3bc8ccc7b1d1b252de32076d4f20d6e1" exitCode=0
Sep 11 16:44:47.390538 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:47.390105    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9" event={"ID":"ed94223d-c737-452e-a64c-a7ea1a2248aa","Type":"ContainerDied","Data":"bedf610b7cfb08ee1ac40807823ae9cc3bc8ccc7b1d1b252de32076d4f20d6e1"}
Sep 11 16:44:48.396913 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:48.396878    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9" event={"ID":"ed94223d-c737-452e-a64c-a7ea1a2248aa","Type":"ContainerStarted","Data":"3a30777891fee6abcabbea14f25b5a1372763b6af7b3135799157c167f95ae93"}
Sep 11 16:44:48.397303 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:48.396943    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9" event={"ID":"ed94223d-c737-452e-a64c-a7ea1a2248aa","Type":"ContainerStarted","Data":"a108c0baea39ad752b3bb5423058565c435c0e81bd6d8797b333807ff6e4b585"}
Sep 11 16:44:48.397303 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:48.397131    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:48.425152 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:48.425104    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9" podStartSLOduration=2.556822915 podStartE2EDuration="6.425088531s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.868265616s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:48.423109468 +0000 UTC m=+1693.763203514" watchObservedRunningTime="2026-09-11 16:44:48.425088531 +0000 UTC m=+1693.765182579"
Sep 11 16:44:49.401546 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:49.401514    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:44:55.412053 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:44:55.412023    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:45:25.440552 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:25.440503    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9" podUID="ed94223d-c737-452e-a64c-a7ea1a2248aa" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 11 16:45:35.415042 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:35.415007    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:45:42.165455 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:42.165421    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"]
Sep 11 16:45:42.165872 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:42.165657    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9" podUID="ed94223d-c737-452e-a64c-a7ea1a2248aa" containerName="kserve-container" containerID="cri-o://a108c0baea39ad752b3bb5423058565c435c0e81bd6d8797b333807ff6e4b585" gracePeriod=30
Sep 11 16:45:42.165872 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:42.165700    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9" podUID="ed94223d-c737-452e-a64c-a7ea1a2248aa" containerName="kube-rbac-proxy" containerID="cri-o://3a30777891fee6abcabbea14f25b5a1372763b6af7b3135799157c167f95ae93" gracePeriod=30
Sep 11 16:45:42.667154 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:42.667123    2561 generic.go:356] "Generic (PLEG): container finished" podID="ed94223d-c737-452e-a64c-a7ea1a2248aa" containerID="3a30777891fee6abcabbea14f25b5a1372763b6af7b3135799157c167f95ae93" exitCode=2
Sep 11 16:45:42.667323 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:42.667195    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9" event={"ID":"ed94223d-c737-452e-a64c-a7ea1a2248aa","Type":"ContainerDied","Data":"3a30777891fee6abcabbea14f25b5a1372763b6af7b3135799157c167f95ae93"}
Sep 11 16:45:45.407435 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:45.407391    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9" podUID="ed94223d-c737-452e-a64c-a7ea1a2248aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 11 16:45:49.704894 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.704848    2561 generic.go:356] "Generic (PLEG): container finished" podID="ed94223d-c737-452e-a64c-a7ea1a2248aa" containerID="a108c0baea39ad752b3bb5423058565c435c0e81bd6d8797b333807ff6e4b585" exitCode=0
Sep 11 16:45:49.705156 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.704911    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9" event={"ID":"ed94223d-c737-452e-a64c-a7ea1a2248aa","Type":"ContainerDied","Data":"a108c0baea39ad752b3bb5423058565c435c0e81bd6d8797b333807ff6e4b585"}
Sep 11 16:45:49.720084 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.720056    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:45:49.826466 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.826377    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed94223d-c737-452e-a64c-a7ea1a2248aa-kserve-provision-location\") pod \"ed94223d-c737-452e-a64c-a7ea1a2248aa\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") "
Sep 11 16:45:49.826466 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.826417    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed94223d-c737-452e-a64c-a7ea1a2248aa-cabundle-cert\") pod \"ed94223d-c737-452e-a64c-a7ea1a2248aa\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") "
Sep 11 16:45:49.826466 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.826454    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed94223d-c737-452e-a64c-a7ea1a2248aa-kube-rbac-proxy-sar-config\") pod \"ed94223d-c737-452e-a64c-a7ea1a2248aa\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") "
Sep 11 16:45:49.826756 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.826496    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px9kf\" (UniqueName: \"kubernetes.io/projected/ed94223d-c737-452e-a64c-a7ea1a2248aa-kube-api-access-px9kf\") pod \"ed94223d-c737-452e-a64c-a7ea1a2248aa\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") "
Sep 11 16:45:49.826756 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.826523    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed94223d-c737-452e-a64c-a7ea1a2248aa-proxy-tls\") pod \"ed94223d-c737-452e-a64c-a7ea1a2248aa\" (UID: \"ed94223d-c737-452e-a64c-a7ea1a2248aa\") "
Sep 11 16:45:49.826873 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.826786    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed94223d-c737-452e-a64c-a7ea1a2248aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed94223d-c737-452e-a64c-a7ea1a2248aa" (UID: "ed94223d-c737-452e-a64c-a7ea1a2248aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:45:49.826927 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.826871    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed94223d-c737-452e-a64c-a7ea1a2248aa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ed94223d-c737-452e-a64c-a7ea1a2248aa" (UID: "ed94223d-c737-452e-a64c-a7ea1a2248aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:45:49.826927 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.826888    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed94223d-c737-452e-a64c-a7ea1a2248aa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ed94223d-c737-452e-a64c-a7ea1a2248aa" (UID: "ed94223d-c737-452e-a64c-a7ea1a2248aa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:45:49.828730 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.828704    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed94223d-c737-452e-a64c-a7ea1a2248aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed94223d-c737-452e-a64c-a7ea1a2248aa" (UID: "ed94223d-c737-452e-a64c-a7ea1a2248aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:45:49.828730 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.828720    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed94223d-c737-452e-a64c-a7ea1a2248aa-kube-api-access-px9kf" (OuterVolumeSpecName: "kube-api-access-px9kf") pod "ed94223d-c737-452e-a64c-a7ea1a2248aa" (UID: "ed94223d-c737-452e-a64c-a7ea1a2248aa"). InnerVolumeSpecName "kube-api-access-px9kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:45:49.927592 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.927553    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed94223d-c737-452e-a64c-a7ea1a2248aa-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:45:49.927592 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.927586    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed94223d-c737-452e-a64c-a7ea1a2248aa-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:45:49.927592 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.927597    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed94223d-c737-452e-a64c-a7ea1a2248aa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:45:49.927820 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.927605    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px9kf\" (UniqueName: \"kubernetes.io/projected/ed94223d-c737-452e-a64c-a7ea1a2248aa-kube-api-access-px9kf\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:45:49.927820 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:49.927615    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed94223d-c737-452e-a64c-a7ea1a2248aa-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:45:50.711990 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:50.711963    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"
Sep 11 16:45:50.712447 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:50.711965    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9" event={"ID":"ed94223d-c737-452e-a64c-a7ea1a2248aa","Type":"ContainerDied","Data":"097f5172dc2a8ba42654ebe24b80c5aa5133a4888451035914456dcfe45a29bc"}
Sep 11 16:45:50.712447 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:50.712123    2561 scope.go:117] "RemoveContainer" containerID="3a30777891fee6abcabbea14f25b5a1372763b6af7b3135799157c167f95ae93"
Sep 11 16:45:50.721489 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:50.721468    2561 scope.go:117] "RemoveContainer" containerID="a108c0baea39ad752b3bb5423058565c435c0e81bd6d8797b333807ff6e4b585"
Sep 11 16:45:50.729696 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:50.729674    2561 scope.go:117] "RemoveContainer" containerID="bedf610b7cfb08ee1ac40807823ae9cc3bc8ccc7b1d1b252de32076d4f20d6e1"
Sep 11 16:45:50.739037 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:50.739014    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"]
Sep 11 16:45:50.745020 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:50.744994    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s6xc9"]
Sep 11 16:45:51.321572 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:45:51.321536    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed94223d-c737-452e-a64c-a7ea1a2248aa" path="/var/lib/kubelet/pods/ed94223d-c737-452e-a64c-a7ea1a2248aa/volumes"
Sep 11 16:46:35.277900 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:46:35.277872    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:46:35.397252 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:46:35.397223    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:47:03.951824 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.951787    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"]
Sep 11 16:47:03.952437 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.952160    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed94223d-c737-452e-a64c-a7ea1a2248aa" containerName="storage-initializer"
Sep 11 16:47:03.952437 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.952187    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed94223d-c737-452e-a64c-a7ea1a2248aa" containerName="storage-initializer"
Sep 11 16:47:03.952437 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.952206    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed94223d-c737-452e-a64c-a7ea1a2248aa" containerName="kserve-container"
Sep 11 16:47:03.952437 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.952214    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed94223d-c737-452e-a64c-a7ea1a2248aa" containerName="kserve-container"
Sep 11 16:47:03.952437 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.952221    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed94223d-c737-452e-a64c-a7ea1a2248aa" containerName="kube-rbac-proxy"
Sep 11 16:47:03.952437 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.952228    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed94223d-c737-452e-a64c-a7ea1a2248aa" containerName="kube-rbac-proxy"
Sep 11 16:47:03.952437 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.952309    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed94223d-c737-452e-a64c-a7ea1a2248aa" containerName="kube-rbac-proxy"
Sep 11 16:47:03.952437 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.952319    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed94223d-c737-452e-a64c-a7ea1a2248aa" containerName="kserve-container"
Sep 11 16:47:03.956723 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.956704    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:03.958907 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.958881    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 11 16:47:03.959020 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.958919    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:47:03.959584 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.959566    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:47:03.960433 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.960417    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 16:47:03.960852 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.960835    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:47:03.961139 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.961121    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 11 16:47:03.961266 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.961248    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:47:03.974392 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:03.974363    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"]
Sep 11 16:47:04.060263 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.060227    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-zrzwc\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:04.060463 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.060287    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xmh\" (UniqueName: \"kubernetes.io/projected/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kube-api-access-n5xmh\") pod \"isvc-canary-create-predictor-6474b875b9-zrzwc\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:04.060463 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.060336    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-zrzwc\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:04.060463 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.060353    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-zrzwc\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:04.060617 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.060474    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-zrzwc\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:04.161903 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.161864    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-zrzwc\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:04.162085 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.161920    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-zrzwc\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:04.162085 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.161964    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xmh\" (UniqueName: \"kubernetes.io/projected/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kube-api-access-n5xmh\") pod \"isvc-canary-create-predictor-6474b875b9-zrzwc\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:04.162085 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.161989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-zrzwc\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:04.162085 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.162012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-zrzwc\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:04.162397 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.162375    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-zrzwc\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:04.162655 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.162632    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-zrzwc\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:04.162754 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.162675    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-zrzwc\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:04.164551 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.164525    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-zrzwc\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:04.172802 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.172771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xmh\" (UniqueName: \"kubernetes.io/projected/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kube-api-access-n5xmh\") pod \"isvc-canary-create-predictor-6474b875b9-zrzwc\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:04.274058 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.273970    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:04.562752 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:04.562723    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"]
Sep 11 16:47:04.564757 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:47:04.564724    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb461b9_6cc4_4862_b57d_d115e3b1c1fd.slice/crio-8e1b838b6f951952d7130bf5642fe84d491ab80ac644744bb3bb4991bc8c2db6 WatchSource:0}: Error finding container 8e1b838b6f951952d7130bf5642fe84d491ab80ac644744bb3bb4991bc8c2db6: Status 404 returned error can't find the container with id 8e1b838b6f951952d7130bf5642fe84d491ab80ac644744bb3bb4991bc8c2db6
Sep 11 16:47:05.060136 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:05.060096    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" event={"ID":"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd","Type":"ContainerStarted","Data":"b36469d3c54a94644b2e0246ea9bfd0200c4519d16effcf801574b693c11f54f"}
Sep 11 16:47:05.060553 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:05.060170    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" event={"ID":"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd","Type":"ContainerStarted","Data":"8e1b838b6f951952d7130bf5642fe84d491ab80ac644744bb3bb4991bc8c2db6"}
Sep 11 16:47:09.080156 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:09.080121    2561 generic.go:356] "Generic (PLEG): container finished" podID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerID="b36469d3c54a94644b2e0246ea9bfd0200c4519d16effcf801574b693c11f54f" exitCode=0
Sep 11 16:47:09.080653 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:09.080169    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" event={"ID":"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd","Type":"ContainerDied","Data":"b36469d3c54a94644b2e0246ea9bfd0200c4519d16effcf801574b693c11f54f"}
Sep 11 16:47:09.081363 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:09.081342    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:47:23.187425 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:23.187385    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" event={"ID":"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd","Type":"ContainerStarted","Data":"d0e54e0fa770a098813601a557e89db26fe2619ff4a8520c5e76f857b11dd3e9"}
Sep 11 16:47:23.187799 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:23.187477    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" event={"ID":"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd","Type":"ContainerStarted","Data":"84824312dc1ebf7a921c2f0d69982f5d5ea7c6b0087f556944ffab4b4ef3e9a9"}
Sep 11 16:47:23.187842 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:23.187809    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:23.214950 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:23.214899    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" podStartSLOduration=2.887971656 podStartE2EDuration="20.214882421s" totalImagesPullingTime="13.70030282s" totalInitContainerRuntime="3.626607945s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:47:23.21481493 +0000 UTC m=+1848.554908975" watchObservedRunningTime="2026-09-11 16:47:23.214882421 +0000 UTC m=+1848.554976466"
Sep 11 16:47:24.192247 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:24.192208    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:24.193487 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:24.193460    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 11 16:47:25.197560 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:25.197520    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 11 16:47:30.202604 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:30.202569    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:47:30.203236 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:30.203209    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 11 16:47:40.203341 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:40.203294    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 11 16:47:50.203383 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:47:50.203340    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 11 16:48:00.203697 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:00.203653    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 11 16:48:10.203808 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:10.203764    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 11 16:48:20.203379 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:20.203333    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 11 16:48:30.204136 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:30.204092    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 11 16:48:40.203994 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:40.203961    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:48:43.828562 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:43.828520    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"]
Sep 11 16:48:43.828948 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:43.828771    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kserve-container" containerID="cri-o://84824312dc1ebf7a921c2f0d69982f5d5ea7c6b0087f556944ffab4b4ef3e9a9" gracePeriod=30
Sep 11 16:48:43.828948 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:43.828862    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kube-rbac-proxy" containerID="cri-o://d0e54e0fa770a098813601a557e89db26fe2619ff4a8520c5e76f857b11dd3e9" gracePeriod=30
Sep 11 16:48:44.572746 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:44.572712    2561 generic.go:356] "Generic (PLEG): container finished" podID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerID="d0e54e0fa770a098813601a557e89db26fe2619ff4a8520c5e76f857b11dd3e9" exitCode=2
Sep 11 16:48:44.572915 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:44.572765    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" event={"ID":"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd","Type":"ContainerDied","Data":"d0e54e0fa770a098813601a557e89db26fe2619ff4a8520c5e76f857b11dd3e9"}
Sep 11 16:48:45.198697 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:45.198654    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 11 16:48:48.576983 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.576958    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:48:48.592857 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.592822    2561 generic.go:356] "Generic (PLEG): container finished" podID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerID="84824312dc1ebf7a921c2f0d69982f5d5ea7c6b0087f556944ffab4b4ef3e9a9" exitCode=0
Sep 11 16:48:48.592998 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.592877    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"
Sep 11 16:48:48.592998 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.592902    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" event={"ID":"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd","Type":"ContainerDied","Data":"84824312dc1ebf7a921c2f0d69982f5d5ea7c6b0087f556944ffab4b4ef3e9a9"}
Sep 11 16:48:48.592998 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.592995    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc" event={"ID":"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd","Type":"ContainerDied","Data":"8e1b838b6f951952d7130bf5642fe84d491ab80ac644744bb3bb4991bc8c2db6"}
Sep 11 16:48:48.593110 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.593026    2561 scope.go:117] "RemoveContainer" containerID="d0e54e0fa770a098813601a557e89db26fe2619ff4a8520c5e76f857b11dd3e9"
Sep 11 16:48:48.602343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.602305    2561 scope.go:117] "RemoveContainer" containerID="84824312dc1ebf7a921c2f0d69982f5d5ea7c6b0087f556944ffab4b4ef3e9a9"
Sep 11 16:48:48.610787 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.610766    2561 scope.go:117] "RemoveContainer" containerID="b36469d3c54a94644b2e0246ea9bfd0200c4519d16effcf801574b693c11f54f"
Sep 11 16:48:48.614016 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.613993    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kube-rbac-proxy-sar-config\") pod \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") "
Sep 11 16:48:48.614109 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.614029    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-cabundle-cert\") pod \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") "
Sep 11 16:48:48.614109 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.614071    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kserve-provision-location\") pod \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") "
Sep 11 16:48:48.614109 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.614094    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-proxy-tls\") pod \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") "
Sep 11 16:48:48.614314 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.614164    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5xmh\" (UniqueName: \"kubernetes.io/projected/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kube-api-access-n5xmh\") pod \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\" (UID: \"aeb461b9-6cc4-4862-b57d-d115e3b1c1fd\") "
Sep 11 16:48:48.614475 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.614449    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" (UID: "aeb461b9-6cc4-4862-b57d-d115e3b1c1fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:48:48.614546 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.614529    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" (UID: "aeb461b9-6cc4-4862-b57d-d115e3b1c1fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:48:48.614587 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.614549    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" (UID: "aeb461b9-6cc4-4862-b57d-d115e3b1c1fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:48:48.616360 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.616334    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" (UID: "aeb461b9-6cc4-4862-b57d-d115e3b1c1fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:48:48.616645 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.616624    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kube-api-access-n5xmh" (OuterVolumeSpecName: "kube-api-access-n5xmh") pod "aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" (UID: "aeb461b9-6cc4-4862-b57d-d115e3b1c1fd"). InnerVolumeSpecName "kube-api-access-n5xmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:48:48.619278 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.619255    2561 scope.go:117] "RemoveContainer" containerID="d0e54e0fa770a098813601a557e89db26fe2619ff4a8520c5e76f857b11dd3e9"
Sep 11 16:48:48.619637 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:48:48.619614    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e54e0fa770a098813601a557e89db26fe2619ff4a8520c5e76f857b11dd3e9\": container with ID starting with d0e54e0fa770a098813601a557e89db26fe2619ff4a8520c5e76f857b11dd3e9 not found: ID does not exist" containerID="d0e54e0fa770a098813601a557e89db26fe2619ff4a8520c5e76f857b11dd3e9"
Sep 11 16:48:48.619696 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.619647    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e54e0fa770a098813601a557e89db26fe2619ff4a8520c5e76f857b11dd3e9"} err="failed to get container status \"d0e54e0fa770a098813601a557e89db26fe2619ff4a8520c5e76f857b11dd3e9\": rpc error: code = NotFound desc = could not find container \"d0e54e0fa770a098813601a557e89db26fe2619ff4a8520c5e76f857b11dd3e9\": container with ID starting with d0e54e0fa770a098813601a557e89db26fe2619ff4a8520c5e76f857b11dd3e9 not found: ID does not exist"
Sep 11 16:48:48.619696 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.619666    2561 scope.go:117] "RemoveContainer" containerID="84824312dc1ebf7a921c2f0d69982f5d5ea7c6b0087f556944ffab4b4ef3e9a9"
Sep 11 16:48:48.619937 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:48:48.619923    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84824312dc1ebf7a921c2f0d69982f5d5ea7c6b0087f556944ffab4b4ef3e9a9\": container with ID starting with 84824312dc1ebf7a921c2f0d69982f5d5ea7c6b0087f556944ffab4b4ef3e9a9 not found: ID does not exist" containerID="84824312dc1ebf7a921c2f0d69982f5d5ea7c6b0087f556944ffab4b4ef3e9a9"
Sep 11 16:48:48.619979 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.619941    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84824312dc1ebf7a921c2f0d69982f5d5ea7c6b0087f556944ffab4b4ef3e9a9"} err="failed to get container status \"84824312dc1ebf7a921c2f0d69982f5d5ea7c6b0087f556944ffab4b4ef3e9a9\": rpc error: code = NotFound desc = could not find container \"84824312dc1ebf7a921c2f0d69982f5d5ea7c6b0087f556944ffab4b4ef3e9a9\": container with ID starting with 84824312dc1ebf7a921c2f0d69982f5d5ea7c6b0087f556944ffab4b4ef3e9a9 not found: ID does not exist"
Sep 11 16:48:48.619979 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.619954    2561 scope.go:117] "RemoveContainer" containerID="b36469d3c54a94644b2e0246ea9bfd0200c4519d16effcf801574b693c11f54f"
Sep 11 16:48:48.620235 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:48:48.620206    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36469d3c54a94644b2e0246ea9bfd0200c4519d16effcf801574b693c11f54f\": container with ID starting with b36469d3c54a94644b2e0246ea9bfd0200c4519d16effcf801574b693c11f54f not found: ID does not exist" containerID="b36469d3c54a94644b2e0246ea9bfd0200c4519d16effcf801574b693c11f54f"
Sep 11 16:48:48.620292 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.620253    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36469d3c54a94644b2e0246ea9bfd0200c4519d16effcf801574b693c11f54f"} err="failed to get container status \"b36469d3c54a94644b2e0246ea9bfd0200c4519d16effcf801574b693c11f54f\": rpc error: code = NotFound desc = could not find container \"b36469d3c54a94644b2e0246ea9bfd0200c4519d16effcf801574b693c11f54f\": container with ID starting with b36469d3c54a94644b2e0246ea9bfd0200c4519d16effcf801574b693c11f54f not found: ID does not exist"
Sep 11 16:48:48.715887 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.715846    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5xmh\" (UniqueName: \"kubernetes.io/projected/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kube-api-access-n5xmh\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:48:48.715887 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.715880    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:48:48.715887 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.715891    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:48:48.715887 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.715899    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:48:48.716154 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.715908    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:48:48.920906 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.920872    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"]
Sep 11 16:48:48.925345 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:48.925310    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-zrzwc"]
Sep 11 16:48:49.204141 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.204034    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"]
Sep 11 16:48:49.204647 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.204629    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="storage-initializer"
Sep 11 16:48:49.204699 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.204650    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="storage-initializer"
Sep 11 16:48:49.204699 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.204663    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kube-rbac-proxy"
Sep 11 16:48:49.204699 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.204671    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kube-rbac-proxy"
Sep 11 16:48:49.204699 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.204683    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kserve-container"
Sep 11 16:48:49.204699 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.204691    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kserve-container"
Sep 11 16:48:49.204854 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.204779    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kserve-container"
Sep 11 16:48:49.204854 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.204795    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" containerName="kube-rbac-proxy"
Sep 11 16:48:49.209521 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.209501    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.211638 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.211597    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 16:48:49.211638 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.211636    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 11 16:48:49.211857 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.211835    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:48:49.212366 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.212345    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:48:49.212460 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.212375    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:48:49.212460 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.212423    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:48:49.212460 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.212424    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 11 16:48:49.219842 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.219812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dd29238-824d-42fa-a044-7ed9a5e43e5e-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-l6qgw\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.219940 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.219863    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dd29238-824d-42fa-a044-7ed9a5e43e5e-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-l6qgw\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.219940 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.219891    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-l6qgw\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.219940 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.219922    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd52f\" (UniqueName: \"kubernetes.io/projected/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kube-api-access-zd52f\") pod \"isvc-canary-promote-predictor-7cf78f884f-l6qgw\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.220047 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.219960    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-l6qgw\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.220214 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.220192    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"]
Sep 11 16:48:49.320548 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.320512    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dd29238-824d-42fa-a044-7ed9a5e43e5e-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-l6qgw\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.320754 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.320557    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dd29238-824d-42fa-a044-7ed9a5e43e5e-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-l6qgw\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.320754 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.320680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-l6qgw\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.320754 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.320726    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd52f\" (UniqueName: \"kubernetes.io/projected/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kube-api-access-zd52f\") pod \"isvc-canary-promote-predictor-7cf78f884f-l6qgw\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.320920 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.320765    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-l6qgw\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.321119 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.321095    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-l6qgw\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.321341 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.321324    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dd29238-824d-42fa-a044-7ed9a5e43e5e-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-l6qgw\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.321597 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.321571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-l6qgw\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.321890 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.321870    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeb461b9-6cc4-4862-b57d-d115e3b1c1fd" path="/var/lib/kubelet/pods/aeb461b9-6cc4-4862-b57d-d115e3b1c1fd/volumes"
Sep 11 16:48:49.323143 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.323126    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dd29238-824d-42fa-a044-7ed9a5e43e5e-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-l6qgw\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.330446 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.330424    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd52f\" (UniqueName: \"kubernetes.io/projected/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kube-api-access-zd52f\") pod \"isvc-canary-promote-predictor-7cf78f884f-l6qgw\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.521669 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.521586    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:49.804882 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:49.804852    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"]
Sep 11 16:48:49.806730 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:48:49.806701    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd29238_824d_42fa_a044_7ed9a5e43e5e.slice/crio-d84dc868b9a8658ce9dd7258d1a16490a10d9302d3fe596bf4698986a0023842 WatchSource:0}: Error finding container d84dc868b9a8658ce9dd7258d1a16490a10d9302d3fe596bf4698986a0023842: Status 404 returned error can't find the container with id d84dc868b9a8658ce9dd7258d1a16490a10d9302d3fe596bf4698986a0023842
Sep 11 16:48:50.605619 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:50.605572    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" event={"ID":"5dd29238-824d-42fa-a044-7ed9a5e43e5e","Type":"ContainerStarted","Data":"cb2eb0c1885ef630489d545c3ba0987b7a9e2f2cb9fc73c22b8bc081ef77267f"}
Sep 11 16:48:50.605792 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:50.605689    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" event={"ID":"5dd29238-824d-42fa-a044-7ed9a5e43e5e","Type":"ContainerStarted","Data":"d84dc868b9a8658ce9dd7258d1a16490a10d9302d3fe596bf4698986a0023842"}
Sep 11 16:48:54.626837 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:54.626802    2561 generic.go:356] "Generic (PLEG): container finished" podID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerID="cb2eb0c1885ef630489d545c3ba0987b7a9e2f2cb9fc73c22b8bc081ef77267f" exitCode=0
Sep 11 16:48:54.627234 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:54.626876    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" event={"ID":"5dd29238-824d-42fa-a044-7ed9a5e43e5e","Type":"ContainerDied","Data":"cb2eb0c1885ef630489d545c3ba0987b7a9e2f2cb9fc73c22b8bc081ef77267f"}
Sep 11 16:48:55.634843 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:55.634799    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" event={"ID":"5dd29238-824d-42fa-a044-7ed9a5e43e5e","Type":"ContainerStarted","Data":"e1ef8bfea91f35be4d23e637ff4e099a9be40df7e74404feed181a9b02fb1605"}
Sep 11 16:48:55.635271 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:55.634863    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" event={"ID":"5dd29238-824d-42fa-a044-7ed9a5e43e5e","Type":"ContainerStarted","Data":"8b61b83ee49cb5637723906cff0bea3b972ad3bf884ca57c0d93c63a28571134"}
Sep 11 16:48:55.635271 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:55.635153    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:55.635271 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:55.635241    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:48:55.636617 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:55.636588    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 11 16:48:55.659924 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:55.659873    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" podStartSLOduration=2.722282834 podStartE2EDuration="6.659860274s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.93757744s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:48:55.659624059 +0000 UTC m=+1940.999718105" watchObservedRunningTime="2026-09-11 16:48:55.659860274 +0000 UTC m=+1940.999954381"
Sep 11 16:48:56.640193 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:48:56.640147    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 11 16:49:01.644740 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:01.644709    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:49:01.645301 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:01.645268    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 11 16:49:11.645377 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:11.645333    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 11 16:49:21.645689 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:21.645649    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 11 16:49:31.645977 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:31.645931    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 11 16:49:41.645965 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:41.645916    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 11 16:49:44.822432 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.822397    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"]
Sep 11 16:49:44.827296 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.827270    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:44.829542 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.829519    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 11 16:49:44.830657 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.830635    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 11 16:49:44.841149 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.841124    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"]
Sep 11 16:49:44.895546 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.895513    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6fp45\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:44.895733 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.895584    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spjgc\" (UniqueName: \"kubernetes.io/projected/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kube-api-access-spjgc\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6fp45\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:44.895733 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.895618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6fp45\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:44.895733 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.895645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6fp45\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:44.895733 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.895701    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6fp45\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:44.996532 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.996493    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6fp45\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:44.996728 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.996579    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spjgc\" (UniqueName: \"kubernetes.io/projected/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kube-api-access-spjgc\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6fp45\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:44.996728 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.996619    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6fp45\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:44.996728 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.996653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6fp45\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:44.996728 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.996685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6fp45\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:44.996949 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.996922    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6fp45\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:44.997285 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.997263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6fp45\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:44.997377 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.997310    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6fp45\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:44.999133 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:44.999114    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6fp45\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:45.013821 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:45.013793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spjgc\" (UniqueName: \"kubernetes.io/projected/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kube-api-access-spjgc\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6fp45\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:45.139333 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:45.139239    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:45.420688 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:45.420657    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"]
Sep 11 16:49:45.423467 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:49:45.423437    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd91363_c9af_4fe4_9cdd_8890c7f3c64a.slice/crio-f253b600db73f3ee09c182102c7480cd64dae18f224acda2b2786b5b5f3fbf1b WatchSource:0}: Error finding container f253b600db73f3ee09c182102c7480cd64dae18f224acda2b2786b5b5f3fbf1b: Status 404 returned error can't find the container with id f253b600db73f3ee09c182102c7480cd64dae18f224acda2b2786b5b5f3fbf1b
Sep 11 16:49:45.883377 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:45.883283    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" event={"ID":"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a","Type":"ContainerStarted","Data":"9dc048f06f6b8b56a9f6bf7af43f93b6e5700c9ada394539197a2d440bb030c7"}
Sep 11 16:49:45.883377 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:45.883347    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" event={"ID":"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a","Type":"ContainerStarted","Data":"f253b600db73f3ee09c182102c7480cd64dae18f224acda2b2786b5b5f3fbf1b"}
Sep 11 16:49:50.909127 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:50.909092    2561 generic.go:356] "Generic (PLEG): container finished" podID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerID="9dc048f06f6b8b56a9f6bf7af43f93b6e5700c9ada394539197a2d440bb030c7" exitCode=0
Sep 11 16:49:50.909623 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:50.909195    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" event={"ID":"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a","Type":"ContainerDied","Data":"9dc048f06f6b8b56a9f6bf7af43f93b6e5700c9ada394539197a2d440bb030c7"}
Sep 11 16:49:51.645661 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:51.645608    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 11 16:49:54.935859 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:54.935823    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" event={"ID":"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a","Type":"ContainerStarted","Data":"c30c3bbedccbaf1151d98e68b2690a0e68a574cfc9b7a12cb45d0ce49af1a6b0"}
Sep 11 16:49:54.936206 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:54.935884    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" event={"ID":"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a","Type":"ContainerStarted","Data":"7a59724244c1f02d8a3e068febe2272af548060efbc8d36634e65bcdbf3432c7"}
Sep 11 16:49:54.936206 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:54.936156    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:54.936343 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:54.936325    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:49:54.937678 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:54.937652    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 11 16:49:54.989149 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:54.983219    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" podStartSLOduration=2.15803774 podStartE2EDuration="10.983200198s" totalImagesPullingTime="3.873283613s" totalInitContainerRuntime="4.951878845s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:49:54.982891537 +0000 UTC m=+2000.322985584" watchObservedRunningTime="2026-09-11 16:49:54.983200198 +0000 UTC m=+2000.323294243"
Sep 11 16:49:55.940729 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:49:55.940682    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 11 16:50:00.947441 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:00.947401    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:50:00.948001 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:00.947965    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 11 16:50:01.645263 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:01.645219    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 11 16:50:10.948510 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:10.948480    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:50:11.645345 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:11.645310    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:50:24.125106 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.125070    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"]
Sep 11 16:50:24.125534 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.125346    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kserve-container" containerID="cri-o://8b61b83ee49cb5637723906cff0bea3b972ad3bf884ca57c0d93c63a28571134" gracePeriod=30
Sep 11 16:50:24.125534 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.125371    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kube-rbac-proxy" containerID="cri-o://e1ef8bfea91f35be4d23e637ff4e099a9be40df7e74404feed181a9b02fb1605" gracePeriod=30
Sep 11 16:50:24.262228 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.262191    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"]
Sep 11 16:50:24.265725 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.265705    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.268291 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.268262    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 11 16:50:24.278122 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.278089    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"]
Sep 11 16:50:24.447149 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.447106    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-8v72b\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.447387 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.447227    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-8v72b\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.447387 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.447299    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-8v72b\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.447387 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.447349    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-8v72b\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.447387 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.447382    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l6gg\" (UniqueName: \"kubernetes.io/projected/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kube-api-access-4l6gg\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-8v72b\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.547875 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.547831    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l6gg\" (UniqueName: \"kubernetes.io/projected/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kube-api-access-4l6gg\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-8v72b\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.547875 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.547882    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-8v72b\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.548133 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.547937    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-8v72b\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.548133 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.547983    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-8v72b\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.548133 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.548014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-8v72b\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.548529 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.548495    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-8v72b\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.548748 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.548726    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-8v72b\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.548825 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.548773    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-8v72b\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.550984 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.550963    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-8v72b\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.558110 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.558085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l6gg\" (UniqueName: \"kubernetes.io/projected/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kube-api-access-4l6gg\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-8v72b\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.577072 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.577041    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:24.878298 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:24.878218    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"]
Sep 11 16:50:24.880612 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:50:24.880584    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aab4a77_7ee4_4655_b8d1_fd966ddd8214.slice/crio-b12fb4f4d804f6fa3ef6f346bec93a6a5e0303931b3f34f6075ad6d6daa3433b WatchSource:0}: Error finding container b12fb4f4d804f6fa3ef6f346bec93a6a5e0303931b3f34f6075ad6d6daa3433b: Status 404 returned error can't find the container with id b12fb4f4d804f6fa3ef6f346bec93a6a5e0303931b3f34f6075ad6d6daa3433b
Sep 11 16:50:25.083868 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:25.083823    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" event={"ID":"6aab4a77-7ee4-4655-b8d1-fd966ddd8214","Type":"ContainerStarted","Data":"4443d070f6a1681ab227a131289baee56be5830b9ddaa685372ef7cfeee8502c"}
Sep 11 16:50:25.084088 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:25.083903    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" event={"ID":"6aab4a77-7ee4-4655-b8d1-fd966ddd8214","Type":"ContainerStarted","Data":"b12fb4f4d804f6fa3ef6f346bec93a6a5e0303931b3f34f6075ad6d6daa3433b"}
Sep 11 16:50:25.086310 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:25.086281    2561 generic.go:356] "Generic (PLEG): container finished" podID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerID="e1ef8bfea91f35be4d23e637ff4e099a9be40df7e74404feed181a9b02fb1605" exitCode=2
Sep 11 16:50:25.086393 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:25.086309    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" event={"ID":"5dd29238-824d-42fa-a044-7ed9a5e43e5e","Type":"ContainerDied","Data":"e1ef8bfea91f35be4d23e637ff4e099a9be40df7e74404feed181a9b02fb1605"}
Sep 11 16:50:26.294163 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:26.294129    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"]
Sep 11 16:50:26.294651 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:26.294454    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kserve-container" containerID="cri-o://7a59724244c1f02d8a3e068febe2272af548060efbc8d36634e65bcdbf3432c7" gracePeriod=30
Sep 11 16:50:26.294651 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:26.294532    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kube-rbac-proxy" containerID="cri-o://c30c3bbedccbaf1151d98e68b2690a0e68a574cfc9b7a12cb45d0ce49af1a6b0" gracePeriod=30
Sep 11 16:50:26.640663 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:26.640566    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 11 16:50:27.097515 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:27.097482    2561 generic.go:356] "Generic (PLEG): container finished" podID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerID="c30c3bbedccbaf1151d98e68b2690a0e68a574cfc9b7a12cb45d0ce49af1a6b0" exitCode=2
Sep 11 16:50:27.097694 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:27.097528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" event={"ID":"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a","Type":"ContainerDied","Data":"c30c3bbedccbaf1151d98e68b2690a0e68a574cfc9b7a12cb45d0ce49af1a6b0"}
Sep 11 16:50:29.110751 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.110668    2561 generic.go:356] "Generic (PLEG): container finished" podID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerID="4443d070f6a1681ab227a131289baee56be5830b9ddaa685372ef7cfeee8502c" exitCode=0
Sep 11 16:50:29.111117 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.110746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" event={"ID":"6aab4a77-7ee4-4655-b8d1-fd966ddd8214","Type":"ContainerDied","Data":"4443d070f6a1681ab227a131289baee56be5830b9ddaa685372ef7cfeee8502c"}
Sep 11 16:50:29.113743 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.113677    2561 generic.go:356] "Generic (PLEG): container finished" podID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerID="8b61b83ee49cb5637723906cff0bea3b972ad3bf884ca57c0d93c63a28571134" exitCode=0
Sep 11 16:50:29.113829 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.113747    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" event={"ID":"5dd29238-824d-42fa-a044-7ed9a5e43e5e","Type":"ContainerDied","Data":"8b61b83ee49cb5637723906cff0bea3b972ad3bf884ca57c0d93c63a28571134"}
Sep 11 16:50:29.180630 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.180602    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:50:29.285116 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.285082    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dd29238-824d-42fa-a044-7ed9a5e43e5e-proxy-tls\") pod \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") "
Sep 11 16:50:29.285329 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.285158    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kserve-provision-location\") pod \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") "
Sep 11 16:50:29.285329 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.285220    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kube-rbac-proxy-sar-config\") pod \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") "
Sep 11 16:50:29.285329 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.285235    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dd29238-824d-42fa-a044-7ed9a5e43e5e-cabundle-cert\") pod \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") "
Sep 11 16:50:29.285329 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.285258    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd52f\" (UniqueName: \"kubernetes.io/projected/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kube-api-access-zd52f\") pod \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\" (UID: \"5dd29238-824d-42fa-a044-7ed9a5e43e5e\") "
Sep 11 16:50:29.285603 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.285571    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5dd29238-824d-42fa-a044-7ed9a5e43e5e" (UID: "5dd29238-824d-42fa-a044-7ed9a5e43e5e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:50:29.285603 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.285588    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5dd29238-824d-42fa-a044-7ed9a5e43e5e" (UID: "5dd29238-824d-42fa-a044-7ed9a5e43e5e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:50:29.285603 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.285600    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd29238-824d-42fa-a044-7ed9a5e43e5e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5dd29238-824d-42fa-a044-7ed9a5e43e5e" (UID: "5dd29238-824d-42fa-a044-7ed9a5e43e5e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:50:29.287344 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.287323    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kube-api-access-zd52f" (OuterVolumeSpecName: "kube-api-access-zd52f") pod "5dd29238-824d-42fa-a044-7ed9a5e43e5e" (UID: "5dd29238-824d-42fa-a044-7ed9a5e43e5e"). InnerVolumeSpecName "kube-api-access-zd52f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:50:29.287439 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.287417    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd29238-824d-42fa-a044-7ed9a5e43e5e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5dd29238-824d-42fa-a044-7ed9a5e43e5e" (UID: "5dd29238-824d-42fa-a044-7ed9a5e43e5e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:50:29.386957 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.386846    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:50:29.386957 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.386887    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:50:29.386957 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.386903    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5dd29238-824d-42fa-a044-7ed9a5e43e5e-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:50:29.386957 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.386915    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd52f\" (UniqueName: \"kubernetes.io/projected/5dd29238-824d-42fa-a044-7ed9a5e43e5e-kube-api-access-zd52f\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:50:29.386957 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:29.386929    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dd29238-824d-42fa-a044-7ed9a5e43e5e-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:50:30.120402 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:30.120308    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"
Sep 11 16:50:30.120402 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:30.120317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw" event={"ID":"5dd29238-824d-42fa-a044-7ed9a5e43e5e","Type":"ContainerDied","Data":"d84dc868b9a8658ce9dd7258d1a16490a10d9302d3fe596bf4698986a0023842"}
Sep 11 16:50:30.120897 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:30.120430    2561 scope.go:117] "RemoveContainer" containerID="e1ef8bfea91f35be4d23e637ff4e099a9be40df7e74404feed181a9b02fb1605"
Sep 11 16:50:30.122824 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:30.122788    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" event={"ID":"6aab4a77-7ee4-4655-b8d1-fd966ddd8214","Type":"ContainerStarted","Data":"7931e826d2f4f5e6f3d998f1d13c77c092925f063ba48103eabdf162945498b8"}
Sep 11 16:50:30.122948 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:30.122859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" event={"ID":"6aab4a77-7ee4-4655-b8d1-fd966ddd8214","Type":"ContainerStarted","Data":"be33e48025995f80ce3289553e2b1ebf27363dc3604da57d0bda83be7c01172d"}
Sep 11 16:50:30.123164 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:30.123146    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:30.123368 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:30.123347    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:30.124578 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:30.124555    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 11 16:50:30.129500 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:30.129482    2561 scope.go:117] "RemoveContainer" containerID="8b61b83ee49cb5637723906cff0bea3b972ad3bf884ca57c0d93c63a28571134"
Sep 11 16:50:30.137379 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:30.137360    2561 scope.go:117] "RemoveContainer" containerID="cb2eb0c1885ef630489d545c3ba0987b7a9e2f2cb9fc73c22b8bc081ef77267f"
Sep 11 16:50:30.144980 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:30.144956    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"]
Sep 11 16:50:30.150237 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:30.150215    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-l6qgw"]
Sep 11 16:50:30.173529 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:30.173468    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" podStartSLOduration=2.354395044 podStartE2EDuration="6.173448905s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.819053861s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:50:30.170552932 +0000 UTC m=+2035.510646978" watchObservedRunningTime="2026-09-11 16:50:30.173448905 +0000 UTC m=+2035.513542951"
Sep 11 16:50:30.941778 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:30.941729    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 11 16:50:31.129136 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:31.129096    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 11 16:50:31.321613 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:31.321528    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" path="/var/lib/kubelet/pods/5dd29238-824d-42fa-a044-7ed9a5e43e5e/volumes"
Sep 11 16:50:35.941912 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:35.941857    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 11 16:50:36.134163 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:36.134129    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:50:36.134762 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:36.134730    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 11 16:50:40.941133 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:40.941088    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 11 16:50:40.941573 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:40.941272    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:50:45.941904 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:45.941854    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 11 16:50:46.135224 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:46.135161    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 11 16:50:50.940921 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:50.940868    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 11 16:50:55.941776 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:55.941725    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 11 16:50:56.135137 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:56.135093    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 11 16:50:56.943952 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:56.943920    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:50:57.035045 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.035008    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-proxy-tls\") pod \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") "
Sep 11 16:50:57.035233 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.035059    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-cabundle-cert\") pod \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") "
Sep 11 16:50:57.035233 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.035102    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kserve-provision-location\") pod \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") "
Sep 11 16:50:57.035233 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.035125    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kube-rbac-proxy-sar-config\") pod \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") "
Sep 11 16:50:57.035233 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.035200    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spjgc\" (UniqueName: \"kubernetes.io/projected/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kube-api-access-spjgc\") pod \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\" (UID: \"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a\") "
Sep 11 16:50:57.035530 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.035508    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" (UID: "2dd91363-c9af-4fe4-9cdd-8890c7f3c64a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:50:57.035617 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.035518    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" (UID: "2dd91363-c9af-4fe4-9cdd-8890c7f3c64a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:50:57.037292 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.037261    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kube-api-access-spjgc" (OuterVolumeSpecName: "kube-api-access-spjgc") pod "2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" (UID: "2dd91363-c9af-4fe4-9cdd-8890c7f3c64a"). InnerVolumeSpecName "kube-api-access-spjgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:50:57.037479 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.037460    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" (UID: "2dd91363-c9af-4fe4-9cdd-8890c7f3c64a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:50:57.045582 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.045553    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" (UID: "2dd91363-c9af-4fe4-9cdd-8890c7f3c64a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:50:57.136777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.136689    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:50:57.136777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.136724    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spjgc\" (UniqueName: \"kubernetes.io/projected/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kube-api-access-spjgc\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:50:57.136777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.136734    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:50:57.136777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.136755    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:50:57.136777 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.136765    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:50:57.263302 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.263265    2561 generic.go:356] "Generic (PLEG): container finished" podID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerID="7a59724244c1f02d8a3e068febe2272af548060efbc8d36634e65bcdbf3432c7" exitCode=137
Sep 11 16:50:57.263464 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.263331    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"
Sep 11 16:50:57.263464 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.263323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" event={"ID":"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a","Type":"ContainerDied","Data":"7a59724244c1f02d8a3e068febe2272af548060efbc8d36634e65bcdbf3432c7"}
Sep 11 16:50:57.263536 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.263468    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45" event={"ID":"2dd91363-c9af-4fe4-9cdd-8890c7f3c64a","Type":"ContainerDied","Data":"f253b600db73f3ee09c182102c7480cd64dae18f224acda2b2786b5b5f3fbf1b"}
Sep 11 16:50:57.263536 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.263501    2561 scope.go:117] "RemoveContainer" containerID="c30c3bbedccbaf1151d98e68b2690a0e68a574cfc9b7a12cb45d0ce49af1a6b0"
Sep 11 16:50:57.279008 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.278976    2561 scope.go:117] "RemoveContainer" containerID="7a59724244c1f02d8a3e068febe2272af548060efbc8d36634e65bcdbf3432c7"
Sep 11 16:50:57.287202 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.287161    2561 scope.go:117] "RemoveContainer" containerID="9dc048f06f6b8b56a9f6bf7af43f93b6e5700c9ada394539197a2d440bb030c7"
Sep 11 16:50:57.295051 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.295025    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"]
Sep 11 16:50:57.296301 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.296279    2561 scope.go:117] "RemoveContainer" containerID="c30c3bbedccbaf1151d98e68b2690a0e68a574cfc9b7a12cb45d0ce49af1a6b0"
Sep 11 16:50:57.296612 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:50:57.296585    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30c3bbedccbaf1151d98e68b2690a0e68a574cfc9b7a12cb45d0ce49af1a6b0\": container with ID starting with c30c3bbedccbaf1151d98e68b2690a0e68a574cfc9b7a12cb45d0ce49af1a6b0 not found: ID does not exist" containerID="c30c3bbedccbaf1151d98e68b2690a0e68a574cfc9b7a12cb45d0ce49af1a6b0"
Sep 11 16:50:57.296689 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.296625    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30c3bbedccbaf1151d98e68b2690a0e68a574cfc9b7a12cb45d0ce49af1a6b0"} err="failed to get container status \"c30c3bbedccbaf1151d98e68b2690a0e68a574cfc9b7a12cb45d0ce49af1a6b0\": rpc error: code = NotFound desc = could not find container \"c30c3bbedccbaf1151d98e68b2690a0e68a574cfc9b7a12cb45d0ce49af1a6b0\": container with ID starting with c30c3bbedccbaf1151d98e68b2690a0e68a574cfc9b7a12cb45d0ce49af1a6b0 not found: ID does not exist"
Sep 11 16:50:57.296689 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.296650    2561 scope.go:117] "RemoveContainer" containerID="7a59724244c1f02d8a3e068febe2272af548060efbc8d36634e65bcdbf3432c7"
Sep 11 16:50:57.296979 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:50:57.296957    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a59724244c1f02d8a3e068febe2272af548060efbc8d36634e65bcdbf3432c7\": container with ID starting with 7a59724244c1f02d8a3e068febe2272af548060efbc8d36634e65bcdbf3432c7 not found: ID does not exist" containerID="7a59724244c1f02d8a3e068febe2272af548060efbc8d36634e65bcdbf3432c7"
Sep 11 16:50:57.297038 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.296985    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a59724244c1f02d8a3e068febe2272af548060efbc8d36634e65bcdbf3432c7"} err="failed to get container status \"7a59724244c1f02d8a3e068febe2272af548060efbc8d36634e65bcdbf3432c7\": rpc error: code = NotFound desc = could not find container \"7a59724244c1f02d8a3e068febe2272af548060efbc8d36634e65bcdbf3432c7\": container with ID starting with 7a59724244c1f02d8a3e068febe2272af548060efbc8d36634e65bcdbf3432c7 not found: ID does not exist"
Sep 11 16:50:57.297038 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.297003    2561 scope.go:117] "RemoveContainer" containerID="9dc048f06f6b8b56a9f6bf7af43f93b6e5700c9ada394539197a2d440bb030c7"
Sep 11 16:50:57.297330 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:50:57.297308    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc048f06f6b8b56a9f6bf7af43f93b6e5700c9ada394539197a2d440bb030c7\": container with ID starting with 9dc048f06f6b8b56a9f6bf7af43f93b6e5700c9ada394539197a2d440bb030c7 not found: ID does not exist" containerID="9dc048f06f6b8b56a9f6bf7af43f93b6e5700c9ada394539197a2d440bb030c7"
Sep 11 16:50:57.297398 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.297332    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc048f06f6b8b56a9f6bf7af43f93b6e5700c9ada394539197a2d440bb030c7"} err="failed to get container status \"9dc048f06f6b8b56a9f6bf7af43f93b6e5700c9ada394539197a2d440bb030c7\": rpc error: code = NotFound desc = could not find container \"9dc048f06f6b8b56a9f6bf7af43f93b6e5700c9ada394539197a2d440bb030c7\": container with ID starting with 9dc048f06f6b8b56a9f6bf7af43f93b6e5700c9ada394539197a2d440bb030c7 not found: ID does not exist"
Sep 11 16:50:57.297833 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.297816    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6fp45"]
Sep 11 16:50:57.320752 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:50:57.320698    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" path="/var/lib/kubelet/pods/2dd91363-c9af-4fe4-9cdd-8890c7f3c64a/volumes"
Sep 11 16:51:06.135504 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:06.135460    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 11 16:51:16.135366 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:16.135323    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 11 16:51:26.135137 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:26.135087    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 11 16:51:35.309427 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:35.309401    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:51:35.427260 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:35.427235    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:51:36.134875 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:36.134830    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 11 16:51:46.136214 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:46.136158    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:51:54.384515 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:54.384470    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"]
Sep 11 16:51:54.385105 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:54.384797    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kserve-container" containerID="cri-o://be33e48025995f80ce3289553e2b1ebf27363dc3604da57d0bda83be7c01172d" gracePeriod=30
Sep 11 16:51:54.385105 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:54.384869    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kube-rbac-proxy" containerID="cri-o://7931e826d2f4f5e6f3d998f1d13c77c092925f063ba48103eabdf162945498b8" gracePeriod=30
Sep 11 16:51:54.541412 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:54.541377    2561 generic.go:356] "Generic (PLEG): container finished" podID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerID="7931e826d2f4f5e6f3d998f1d13c77c092925f063ba48103eabdf162945498b8" exitCode=2
Sep 11 16:51:54.541569 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:54.541437    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" event={"ID":"6aab4a77-7ee4-4655-b8d1-fd966ddd8214","Type":"ContainerDied","Data":"7931e826d2f4f5e6f3d998f1d13c77c092925f063ba48103eabdf162945498b8"}
Sep 11 16:51:56.130025 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:56.129968    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 11 16:51:56.135450 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:56.135412    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 11 16:51:59.438977 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.438947    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:51:59.571646 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.571526    2561 generic.go:356] "Generic (PLEG): container finished" podID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerID="be33e48025995f80ce3289553e2b1ebf27363dc3604da57d0bda83be7c01172d" exitCode=0
Sep 11 16:51:59.571646 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.571562    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" event={"ID":"6aab4a77-7ee4-4655-b8d1-fd966ddd8214","Type":"ContainerDied","Data":"be33e48025995f80ce3289553e2b1ebf27363dc3604da57d0bda83be7c01172d"}
Sep 11 16:51:59.571646 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.571608    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"
Sep 11 16:51:59.571646 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.571632    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b" event={"ID":"6aab4a77-7ee4-4655-b8d1-fd966ddd8214","Type":"ContainerDied","Data":"b12fb4f4d804f6fa3ef6f346bec93a6a5e0303931b3f34f6075ad6d6daa3433b"}
Sep 11 16:51:59.571976 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.571672    2561 scope.go:117] "RemoveContainer" containerID="7931e826d2f4f5e6f3d998f1d13c77c092925f063ba48103eabdf162945498b8"
Sep 11 16:51:59.581058 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.581037    2561 scope.go:117] "RemoveContainer" containerID="be33e48025995f80ce3289553e2b1ebf27363dc3604da57d0bda83be7c01172d"
Sep 11 16:51:59.589095 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.589073    2561 scope.go:117] "RemoveContainer" containerID="4443d070f6a1681ab227a131289baee56be5830b9ddaa685372ef7cfeee8502c"
Sep 11 16:51:59.599079 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.599058    2561 scope.go:117] "RemoveContainer" containerID="7931e826d2f4f5e6f3d998f1d13c77c092925f063ba48103eabdf162945498b8"
Sep 11 16:51:59.599413 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:51:59.599392    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7931e826d2f4f5e6f3d998f1d13c77c092925f063ba48103eabdf162945498b8\": container with ID starting with 7931e826d2f4f5e6f3d998f1d13c77c092925f063ba48103eabdf162945498b8 not found: ID does not exist" containerID="7931e826d2f4f5e6f3d998f1d13c77c092925f063ba48103eabdf162945498b8"
Sep 11 16:51:59.599487 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.599424    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7931e826d2f4f5e6f3d998f1d13c77c092925f063ba48103eabdf162945498b8"} err="failed to get container status \"7931e826d2f4f5e6f3d998f1d13c77c092925f063ba48103eabdf162945498b8\": rpc error: code = NotFound desc = could not find container \"7931e826d2f4f5e6f3d998f1d13c77c092925f063ba48103eabdf162945498b8\": container with ID starting with 7931e826d2f4f5e6f3d998f1d13c77c092925f063ba48103eabdf162945498b8 not found: ID does not exist"
Sep 11 16:51:59.599487 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.599444    2561 scope.go:117] "RemoveContainer" containerID="be33e48025995f80ce3289553e2b1ebf27363dc3604da57d0bda83be7c01172d"
Sep 11 16:51:59.599747 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:51:59.599722    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be33e48025995f80ce3289553e2b1ebf27363dc3604da57d0bda83be7c01172d\": container with ID starting with be33e48025995f80ce3289553e2b1ebf27363dc3604da57d0bda83be7c01172d not found: ID does not exist" containerID="be33e48025995f80ce3289553e2b1ebf27363dc3604da57d0bda83be7c01172d"
Sep 11 16:51:59.599803 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.599759    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be33e48025995f80ce3289553e2b1ebf27363dc3604da57d0bda83be7c01172d"} err="failed to get container status \"be33e48025995f80ce3289553e2b1ebf27363dc3604da57d0bda83be7c01172d\": rpc error: code = NotFound desc = could not find container \"be33e48025995f80ce3289553e2b1ebf27363dc3604da57d0bda83be7c01172d\": container with ID starting with be33e48025995f80ce3289553e2b1ebf27363dc3604da57d0bda83be7c01172d not found: ID does not exist"
Sep 11 16:51:59.599803 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.599782    2561 scope.go:117] "RemoveContainer" containerID="4443d070f6a1681ab227a131289baee56be5830b9ddaa685372ef7cfeee8502c"
Sep 11 16:51:59.600067 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:51:59.600048    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4443d070f6a1681ab227a131289baee56be5830b9ddaa685372ef7cfeee8502c\": container with ID starting with 4443d070f6a1681ab227a131289baee56be5830b9ddaa685372ef7cfeee8502c not found: ID does not exist" containerID="4443d070f6a1681ab227a131289baee56be5830b9ddaa685372ef7cfeee8502c"
Sep 11 16:51:59.600125 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.600074    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4443d070f6a1681ab227a131289baee56be5830b9ddaa685372ef7cfeee8502c"} err="failed to get container status \"4443d070f6a1681ab227a131289baee56be5830b9ddaa685372ef7cfeee8502c\": rpc error: code = NotFound desc = could not find container \"4443d070f6a1681ab227a131289baee56be5830b9ddaa685372ef7cfeee8502c\": container with ID starting with 4443d070f6a1681ab227a131289baee56be5830b9ddaa685372ef7cfeee8502c not found: ID does not exist"
Sep 11 16:51:59.606434 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.606411    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-cabundle-cert\") pod \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") "
Sep 11 16:51:59.606537 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.606446    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kserve-provision-location\") pod \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") "
Sep 11 16:51:59.606537 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.606474    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kube-rbac-proxy-sar-config\") pod \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") "
Sep 11 16:51:59.606537 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.606506    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-proxy-tls\") pod \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") "
Sep 11 16:51:59.606693 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.606564    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l6gg\" (UniqueName: \"kubernetes.io/projected/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kube-api-access-4l6gg\") pod \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\" (UID: \"6aab4a77-7ee4-4655-b8d1-fd966ddd8214\") "
Sep 11 16:51:59.606870 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.606848    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6aab4a77-7ee4-4655-b8d1-fd966ddd8214" (UID: "6aab4a77-7ee4-4655-b8d1-fd966ddd8214"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:51:59.607227 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.607200    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6aab4a77-7ee4-4655-b8d1-fd966ddd8214" (UID: "6aab4a77-7ee4-4655-b8d1-fd966ddd8214"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:51:59.607465 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.607442    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6aab4a77-7ee4-4655-b8d1-fd966ddd8214" (UID: "6aab4a77-7ee4-4655-b8d1-fd966ddd8214"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:51:59.608794 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.608762    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6aab4a77-7ee4-4655-b8d1-fd966ddd8214" (UID: "6aab4a77-7ee4-4655-b8d1-fd966ddd8214"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:51:59.608870 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.608826    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kube-api-access-4l6gg" (OuterVolumeSpecName: "kube-api-access-4l6gg") pod "6aab4a77-7ee4-4655-b8d1-fd966ddd8214" (UID: "6aab4a77-7ee4-4655-b8d1-fd966ddd8214"). InnerVolumeSpecName "kube-api-access-4l6gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:51:59.707311 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.707268    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l6gg\" (UniqueName: \"kubernetes.io/projected/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kube-api-access-4l6gg\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:51:59.707311 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.707305    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:51:59.707311 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.707314    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:51:59.707563 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.707323    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:51:59.707563 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.707334    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aab4a77-7ee4-4655-b8d1-fd966ddd8214-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:51:59.768685 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.768650    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"]
Sep 11 16:51:59.769006 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.768993    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kube-rbac-proxy"
Sep 11 16:51:59.769054 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769008    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kube-rbac-proxy"
Sep 11 16:51:59.769054 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769020    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kube-rbac-proxy"
Sep 11 16:51:59.769054 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769025    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kube-rbac-proxy"
Sep 11 16:51:59.769054 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769038    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="storage-initializer"
Sep 11 16:51:59.769054 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769043    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="storage-initializer"
Sep 11 16:51:59.769054 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769051    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="storage-initializer"
Sep 11 16:51:59.769054 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769056    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="storage-initializer"
Sep 11 16:51:59.769322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769063    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="storage-initializer"
Sep 11 16:51:59.769322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769068    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="storage-initializer"
Sep 11 16:51:59.769322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769075    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kserve-container"
Sep 11 16:51:59.769322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769081    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kserve-container"
Sep 11 16:51:59.769322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769089    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kserve-container"
Sep 11 16:51:59.769322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769095    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kserve-container"
Sep 11 16:51:59.769322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769101    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kserve-container"
Sep 11 16:51:59.769322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769107    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kserve-container"
Sep 11 16:51:59.769322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769114    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kube-rbac-proxy"
Sep 11 16:51:59.769322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769119    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kube-rbac-proxy"
Sep 11 16:51:59.769322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769190    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kserve-container"
Sep 11 16:51:59.769322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769199    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kserve-container"
Sep 11 16:51:59.769322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769205    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd91363-c9af-4fe4-9cdd-8890c7f3c64a" containerName="kube-rbac-proxy"
Sep 11 16:51:59.769322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769212    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kube-rbac-proxy"
Sep 11 16:51:59.769322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769217    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" containerName="kserve-container"
Sep 11 16:51:59.769322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.769223    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd29238-824d-42fa-a044-7ed9a5e43e5e" containerName="kube-rbac-proxy"
Sep 11 16:51:59.774028 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.774006    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:51:59.776912 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.776884    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 11 16:51:59.777092 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.777070    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 11 16:51:59.785484 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.785457    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"]
Sep 11 16:51:59.896259 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.896223    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"]
Sep 11 16:51:59.901384 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.901351    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-8v72b"]
Sep 11 16:51:59.908747 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.908727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44f0d988-c82b-4f24-9556-ff1ecd9c03de-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:51:59.908824 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.908794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:51:59.908873 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.908861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq9rv\" (UniqueName: \"kubernetes.io/projected/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kube-api-access-dq9rv\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:51:59.908921 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.908899    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f0d988-c82b-4f24-9556-ff1ecd9c03de-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:51:59.908921 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:51:59.908917    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:52:00.009677 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:00.009629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f0d988-c82b-4f24-9556-ff1ecd9c03de-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:52:00.009677 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:00.009673    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:52:00.009892 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:00.009738    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44f0d988-c82b-4f24-9556-ff1ecd9c03de-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:52:00.009892 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:00.009758    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:52:00.009892 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:00.009804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq9rv\" (UniqueName: \"kubernetes.io/projected/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kube-api-access-dq9rv\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:52:00.010264 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:00.010236    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:52:00.010535 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:00.010513    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44f0d988-c82b-4f24-9556-ff1ecd9c03de-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:52:00.010592 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:00.010525    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:52:00.012169 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:00.012150    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f0d988-c82b-4f24-9556-ff1ecd9c03de-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:52:00.019706 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:00.019673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq9rv\" (UniqueName: \"kubernetes.io/projected/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kube-api-access-dq9rv\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:52:00.085769 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:00.085728    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:52:00.366965 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:00.366931    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"]
Sep 11 16:52:00.369471 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:52:00.369444    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f0d988_c82b_4f24_9556_ff1ecd9c03de.slice/crio-a2fedd6bed5959bd2a97202eebeec928cf33d9d4a6f1ebd96b3020b0b25adbce WatchSource:0}: Error finding container a2fedd6bed5959bd2a97202eebeec928cf33d9d4a6f1ebd96b3020b0b25adbce: Status 404 returned error can't find the container with id a2fedd6bed5959bd2a97202eebeec928cf33d9d4a6f1ebd96b3020b0b25adbce
Sep 11 16:52:00.577609 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:00.577563    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" event={"ID":"44f0d988-c82b-4f24-9556-ff1ecd9c03de","Type":"ContainerStarted","Data":"595dcec535b681b942875d01a4ae221dffe700cf589ba14b6ad9bc0459ceb1d9"}
Sep 11 16:52:00.577988 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:00.577634    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" event={"ID":"44f0d988-c82b-4f24-9556-ff1ecd9c03de","Type":"ContainerStarted","Data":"a2fedd6bed5959bd2a97202eebeec928cf33d9d4a6f1ebd96b3020b0b25adbce"}
Sep 11 16:52:01.321548 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:01.321512    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aab4a77-7ee4-4655-b8d1-fd966ddd8214" path="/var/lib/kubelet/pods/6aab4a77-7ee4-4655-b8d1-fd966ddd8214/volumes"
Sep 11 16:52:04.600382 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:04.600343    2561 generic.go:356] "Generic (PLEG): container finished" podID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerID="595dcec535b681b942875d01a4ae221dffe700cf589ba14b6ad9bc0459ceb1d9" exitCode=0
Sep 11 16:52:04.600768 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:04.600421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" event={"ID":"44f0d988-c82b-4f24-9556-ff1ecd9c03de","Type":"ContainerDied","Data":"595dcec535b681b942875d01a4ae221dffe700cf589ba14b6ad9bc0459ceb1d9"}
Sep 11 16:52:05.608108 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:05.608077    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" event={"ID":"44f0d988-c82b-4f24-9556-ff1ecd9c03de","Type":"ContainerStarted","Data":"b330a131ae29db8c9b14ca29d8c0ae5876b6d4068c9686a2569bb100e6577aaa"}
Sep 11 16:52:05.608528 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:05.608134    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" event={"ID":"44f0d988-c82b-4f24-9556-ff1ecd9c03de","Type":"ContainerStarted","Data":"bafcf6976942367b487a0658b933668d0b2ac25ab4a62475365afb5858343ad4"}
Sep 11 16:52:05.608528 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:05.608422    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:52:05.608611 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:05.608583    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:52:05.610000 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:05.609971    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 11 16:52:05.634438 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:05.634390    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" podStartSLOduration=3.084664287 podStartE2EDuration="6.634375605s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.549711318s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:52:05.632828335 +0000 UTC m=+2130.972922379" watchObservedRunningTime="2026-09-11 16:52:05.634375605 +0000 UTC m=+2130.974469650"
Sep 11 16:52:06.613498 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:06.613456    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 11 16:52:11.617772 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:11.617742    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:52:11.618380 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:11.618299    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 11 16:52:21.618561 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:21.618515    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 11 16:52:31.619216 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:31.619157    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 11 16:52:41.618157 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:41.618114    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 11 16:52:50.563494 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.563452    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"]
Sep 11 16:52:50.568531 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.568500    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:50.570769 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.570746    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 11 16:52:50.571141 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.571114    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 11 16:52:50.580845 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.580816    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"]
Sep 11 16:52:50.759675 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.759636    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-cmnxd\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:50.759848 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.759689    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-cmnxd\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:50.759848 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.759708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-cmnxd\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:50.759848 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.759760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-cmnxd\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:50.759848 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.759791    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64bft\" (UniqueName: \"kubernetes.io/projected/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kube-api-access-64bft\") pod \"isvc-xgboost-predictor-7566d65bf7-cmnxd\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:50.861161 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.861059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-cmnxd\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:50.861161 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.861107    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-cmnxd\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:50.861161 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.861140    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64bft\" (UniqueName: \"kubernetes.io/projected/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kube-api-access-64bft\") pod \"isvc-xgboost-predictor-7566d65bf7-cmnxd\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:50.861477 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.861220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-cmnxd\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:50.861477 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.861252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-cmnxd\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:50.861613 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.861589    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-cmnxd\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:50.861926 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.861905    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-cmnxd\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:50.861976 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.861913    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-cmnxd\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:50.863795 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.863771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-cmnxd\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:50.871807 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.871780    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64bft\" (UniqueName: \"kubernetes.io/projected/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kube-api-access-64bft\") pod \"isvc-xgboost-predictor-7566d65bf7-cmnxd\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:50.880652 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:50.880616    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:52:51.163074 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:51.163048    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"]
Sep 11 16:52:51.165430 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:52:51.165397    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e646f1_7168_451f_8ba7_eddaf3b9bc2d.slice/crio-79c486c3679748ef9fc3e682fa7e21a14501f013a0d8f314dfe98d55aa1be191 WatchSource:0}: Error finding container 79c486c3679748ef9fc3e682fa7e21a14501f013a0d8f314dfe98d55aa1be191: Status 404 returned error can't find the container with id 79c486c3679748ef9fc3e682fa7e21a14501f013a0d8f314dfe98d55aa1be191
Sep 11 16:52:51.619225 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:51.619106    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 11 16:52:51.831491 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:51.831444    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" event={"ID":"72e646f1-7168-451f-8ba7-eddaf3b9bc2d","Type":"ContainerStarted","Data":"b06c36098c973ba5140f8d5b42cb1c98db51b5b7fb4acc2c8bc0751784841657"}
Sep 11 16:52:51.831670 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:51.831558    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" event={"ID":"72e646f1-7168-451f-8ba7-eddaf3b9bc2d","Type":"ContainerStarted","Data":"79c486c3679748ef9fc3e682fa7e21a14501f013a0d8f314dfe98d55aa1be191"}
Sep 11 16:52:55.856221 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:55.856162    2561 generic.go:356] "Generic (PLEG): container finished" podID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerID="b06c36098c973ba5140f8d5b42cb1c98db51b5b7fb4acc2c8bc0751784841657" exitCode=0
Sep 11 16:52:55.856679 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:55.856238    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" event={"ID":"72e646f1-7168-451f-8ba7-eddaf3b9bc2d","Type":"ContainerDied","Data":"b06c36098c973ba5140f8d5b42cb1c98db51b5b7fb4acc2c8bc0751784841657"}
Sep 11 16:52:55.857662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:52:55.857645    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:53:01.618260 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:01.618212    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 11 16:53:11.618321 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:11.618251    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 11 16:53:13.956378 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:13.956337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" event={"ID":"72e646f1-7168-451f-8ba7-eddaf3b9bc2d","Type":"ContainerStarted","Data":"ed27313baa5aae4597c5bc0e3fd201566e9d65cfcffa8d4d675d2dabe3e44b1b"}
Sep 11 16:53:13.956831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:13.956398    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" event={"ID":"72e646f1-7168-451f-8ba7-eddaf3b9bc2d","Type":"ContainerStarted","Data":"406c5521b5941d1c0b1c493fd4d45ecb7b14bf6a1f53fede09f98290bd40837e"}
Sep 11 16:53:13.956831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:13.956687    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:53:13.956831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:13.956732    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:53:13.958061 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:13.958033    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 11 16:53:13.980162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:13.980112    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" podStartSLOduration=2.852758805 podStartE2EDuration="23.980094844s" totalImagesPullingTime="17.294627708s" totalInitContainerRuntime="3.832708331s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:53:13.979534247 +0000 UTC m=+2199.319628294" watchObservedRunningTime="2026-09-11 16:53:13.980094844 +0000 UTC m=+2199.320188890"
Sep 11 16:53:14.962316 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:14.962272    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 11 16:53:19.967322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:19.967292    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:53:19.967789 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:19.967737    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 11 16:53:21.618363 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:21.618322    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 11 16:53:29.967889 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:29.967844    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 11 16:53:31.619816 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:31.619784    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:53:39.968299 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:39.968262    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 11 16:53:49.968167 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:49.968128    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 11 16:53:59.968682 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:53:59.968576    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 11 16:54:09.968128 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:09.968088    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 11 16:54:19.968288 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:19.968258    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:54:20.687988 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:20.687946    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"]
Sep 11 16:54:20.688227 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:20.688203    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="kserve-container" containerID="cri-o://406c5521b5941d1c0b1c493fd4d45ecb7b14bf6a1f53fede09f98290bd40837e" gracePeriod=30
Sep 11 16:54:20.688314 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:20.688237    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="kube-rbac-proxy" containerID="cri-o://ed27313baa5aae4597c5bc0e3fd201566e9d65cfcffa8d4d675d2dabe3e44b1b" gracePeriod=30
Sep 11 16:54:21.293334 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:21.293295    2561 generic.go:356] "Generic (PLEG): container finished" podID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerID="ed27313baa5aae4597c5bc0e3fd201566e9d65cfcffa8d4d675d2dabe3e44b1b" exitCode=2
Sep 11 16:54:21.293719 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:21.293366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" event={"ID":"72e646f1-7168-451f-8ba7-eddaf3b9bc2d","Type":"ContainerDied","Data":"ed27313baa5aae4597c5bc0e3fd201566e9d65cfcffa8d4d675d2dabe3e44b1b"}
Sep 11 16:54:24.312061 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.312025    2561 generic.go:356] "Generic (PLEG): container finished" podID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerID="406c5521b5941d1c0b1c493fd4d45ecb7b14bf6a1f53fede09f98290bd40837e" exitCode=0
Sep 11 16:54:24.312438 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.312097    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" event={"ID":"72e646f1-7168-451f-8ba7-eddaf3b9bc2d","Type":"ContainerDied","Data":"406c5521b5941d1c0b1c493fd4d45ecb7b14bf6a1f53fede09f98290bd40837e"}
Sep 11 16:54:24.333463 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.333440    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:54:24.484112 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.484073    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kube-rbac-proxy-sar-config\") pod \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") "
Sep 11 16:54:24.484330 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.484122    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64bft\" (UniqueName: \"kubernetes.io/projected/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kube-api-access-64bft\") pod \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") "
Sep 11 16:54:24.484330 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.484152    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-proxy-tls\") pod \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") "
Sep 11 16:54:24.484330 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.484190    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-cabundle-cert\") pod \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") "
Sep 11 16:54:24.484330 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.484255    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kserve-provision-location\") pod \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\" (UID: \"72e646f1-7168-451f-8ba7-eddaf3b9bc2d\") "
Sep 11 16:54:24.484603 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.484560    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "72e646f1-7168-451f-8ba7-eddaf3b9bc2d" (UID: "72e646f1-7168-451f-8ba7-eddaf3b9bc2d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:54:24.484676 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.484612    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "72e646f1-7168-451f-8ba7-eddaf3b9bc2d" (UID: "72e646f1-7168-451f-8ba7-eddaf3b9bc2d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:54:24.484676 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.484612    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "72e646f1-7168-451f-8ba7-eddaf3b9bc2d" (UID: "72e646f1-7168-451f-8ba7-eddaf3b9bc2d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:54:24.486407 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.486385    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "72e646f1-7168-451f-8ba7-eddaf3b9bc2d" (UID: "72e646f1-7168-451f-8ba7-eddaf3b9bc2d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:54:24.486486 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.486410    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kube-api-access-64bft" (OuterVolumeSpecName: "kube-api-access-64bft") pod "72e646f1-7168-451f-8ba7-eddaf3b9bc2d" (UID: "72e646f1-7168-451f-8ba7-eddaf3b9bc2d"). InnerVolumeSpecName "kube-api-access-64bft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:54:24.591656 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.586303    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:54:24.591656 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.586358    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:54:24.591656 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.586419    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64bft\" (UniqueName: \"kubernetes.io/projected/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-kube-api-access-64bft\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:54:24.591656 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.586456    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:54:24.591656 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:24.586474    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72e646f1-7168-451f-8ba7-eddaf3b9bc2d-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:54:25.319664 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.319636    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"
Sep 11 16:54:25.321260 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.321233    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd" event={"ID":"72e646f1-7168-451f-8ba7-eddaf3b9bc2d","Type":"ContainerDied","Data":"79c486c3679748ef9fc3e682fa7e21a14501f013a0d8f314dfe98d55aa1be191"}
Sep 11 16:54:25.321353 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.321293    2561 scope.go:117] "RemoveContainer" containerID="ed27313baa5aae4597c5bc0e3fd201566e9d65cfcffa8d4d675d2dabe3e44b1b"
Sep 11 16:54:25.331194 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.331156    2561 scope.go:117] "RemoveContainer" containerID="406c5521b5941d1c0b1c493fd4d45ecb7b14bf6a1f53fede09f98290bd40837e"
Sep 11 16:54:25.338705 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.338689    2561 scope.go:117] "RemoveContainer" containerID="b06c36098c973ba5140f8d5b42cb1c98db51b5b7fb4acc2c8bc0751784841657"
Sep 11 16:54:25.346435 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.346414    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"]
Sep 11 16:54:25.351271 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.351248    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-cmnxd"]
Sep 11 16:54:25.935088 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.935051    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"]
Sep 11 16:54:25.935521 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.935500    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="kserve-container"
Sep 11 16:54:25.935521 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.935517    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="kserve-container"
Sep 11 16:54:25.935662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.935532    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="storage-initializer"
Sep 11 16:54:25.935662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.935541    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="storage-initializer"
Sep 11 16:54:25.935662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.935553    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="kube-rbac-proxy"
Sep 11 16:54:25.935662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.935559    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="kube-rbac-proxy"
Sep 11 16:54:25.935662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.935632    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="kserve-container"
Sep 11 16:54:25.935662 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.935642    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" containerName="kube-rbac-proxy"
Sep 11 16:54:25.939294 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.939275    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:25.943009 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.942978    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 11 16:54:25.943307 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.943285    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 11 16:54:25.958318 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:25.958295    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"]
Sep 11 16:54:26.100645 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.100608    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:26.100645 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.100655    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:26.100921 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.100679    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:26.100921 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.100796    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:26.100921 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.100865    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvdfs\" (UniqueName: \"kubernetes.io/projected/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kube-api-access-cvdfs\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:26.201718 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.201632    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:26.201718 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.201670    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:26.201718 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.201693    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:26.201984 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.201725    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:26.201984 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.201760    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvdfs\" (UniqueName: \"kubernetes.io/projected/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kube-api-access-cvdfs\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:26.202138 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.202116    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:26.202393 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.202374    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:26.202439 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.202387    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:26.204339 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.204318    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:26.212693 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.212670    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvdfs\" (UniqueName: \"kubernetes.io/projected/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kube-api-access-cvdfs\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:26.252538 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.252495    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:26.552516 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:54:26.552481    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c7e7db_9fb4_40e7_8989_6105e9d1e10a.slice/crio-a7108b89b333266ebfe8b7890bd5760b70b3a019a78f78f1dda281e63a9067a9 WatchSource:0}: Error finding container a7108b89b333266ebfe8b7890bd5760b70b3a019a78f78f1dda281e63a9067a9: Status 404 returned error can't find the container with id a7108b89b333266ebfe8b7890bd5760b70b3a019a78f78f1dda281e63a9067a9
Sep 11 16:54:26.552857 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:26.552561    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"]
Sep 11 16:54:27.321928 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:27.321894    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72e646f1-7168-451f-8ba7-eddaf3b9bc2d" path="/var/lib/kubelet/pods/72e646f1-7168-451f-8ba7-eddaf3b9bc2d/volumes"
Sep 11 16:54:27.335539 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:27.335507    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q" event={"ID":"49c7e7db-9fb4-40e7-8989-6105e9d1e10a","Type":"ContainerStarted","Data":"683f49a4792e1694ee1054e343e456bbead6f76b635d19d939c744372c7cd5d9"}
Sep 11 16:54:27.335677 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:27.335567    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q" event={"ID":"49c7e7db-9fb4-40e7-8989-6105e9d1e10a","Type":"ContainerStarted","Data":"a7108b89b333266ebfe8b7890bd5760b70b3a019a78f78f1dda281e63a9067a9"}
Sep 11 16:54:31.357096 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:31.357060    2561 generic.go:356] "Generic (PLEG): container finished" podID="49c7e7db-9fb4-40e7-8989-6105e9d1e10a" containerID="683f49a4792e1694ee1054e343e456bbead6f76b635d19d939c744372c7cd5d9" exitCode=0
Sep 11 16:54:31.357488 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:31.357130    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q" event={"ID":"49c7e7db-9fb4-40e7-8989-6105e9d1e10a","Type":"ContainerDied","Data":"683f49a4792e1694ee1054e343e456bbead6f76b635d19d939c744372c7cd5d9"}
Sep 11 16:54:32.365307 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:32.365269    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q" event={"ID":"49c7e7db-9fb4-40e7-8989-6105e9d1e10a","Type":"ContainerStarted","Data":"b6442749c019ad5a4368613fae9e903683bf9053263e07f32a16f3ef47583289"}
Sep 11 16:54:32.365729 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:32.365336    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q" event={"ID":"49c7e7db-9fb4-40e7-8989-6105e9d1e10a","Type":"ContainerStarted","Data":"f08d9592ede435e6ef31cf8421f91d843b92f1ab5ce4620f5dc96403c48f077d"}
Sep 11 16:54:32.365729 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:32.365556    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:32.365729 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:32.365705    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:54:32.393969 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:32.393917    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q" podStartSLOduration=3.433258565 podStartE2EDuration="7.393902955s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.96064439s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:54:32.392063702 +0000 UTC m=+2277.732157748" watchObservedRunningTime="2026-09-11 16:54:32.393902955 +0000 UTC m=+2277.733997063"
Sep 11 16:54:38.375193 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:54:38.375153    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:55:08.380037 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:08.380001    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:55:16.080927 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:16.080891    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"]
Sep 11 16:55:16.081422 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:16.081168    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q" podUID="49c7e7db-9fb4-40e7-8989-6105e9d1e10a" containerName="kserve-container" containerID="cri-o://f08d9592ede435e6ef31cf8421f91d843b92f1ab5ce4620f5dc96403c48f077d" gracePeriod=30
Sep 11 16:55:16.081422 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:16.081220    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q" podUID="49c7e7db-9fb4-40e7-8989-6105e9d1e10a" containerName="kube-rbac-proxy" containerID="cri-o://b6442749c019ad5a4368613fae9e903683bf9053263e07f32a16f3ef47583289" gracePeriod=30
Sep 11 16:55:16.581408 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:16.581377    2561 generic.go:356] "Generic (PLEG): container finished" podID="49c7e7db-9fb4-40e7-8989-6105e9d1e10a" containerID="b6442749c019ad5a4368613fae9e903683bf9053263e07f32a16f3ef47583289" exitCode=2
Sep 11 16:55:16.581581 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:16.581451    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q" event={"ID":"49c7e7db-9fb4-40e7-8989-6105e9d1e10a","Type":"ContainerDied","Data":"b6442749c019ad5a4368613fae9e903683bf9053263e07f32a16f3ef47583289"}
Sep 11 16:55:18.370628 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:18.370584    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q" podUID="49c7e7db-9fb4-40e7-8989-6105e9d1e10a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 11 16:55:21.217807 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.217769    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"]
Sep 11 16:55:21.221736 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.221711    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.224298 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.224274    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 11 16:55:21.224754 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.224732    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 11 16:55:21.295945 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.295904    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"]
Sep 11 16:55:21.385820 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.385787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-kn69w\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.386007 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.385870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85scq\" (UniqueName: \"kubernetes.io/projected/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kube-api-access-85scq\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-kn69w\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.386007 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.385904    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-kn69w\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.386007 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.385923    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-kn69w\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.386121 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.386010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-kn69w\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.486534 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.486429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-kn69w\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.486534 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.486476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-kn69w\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.486772 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.486543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85scq\" (UniqueName: \"kubernetes.io/projected/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kube-api-access-85scq\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-kn69w\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.486772 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.486704    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-kn69w\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.486772 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.486741    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-kn69w\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.487140 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.487116    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-kn69w\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.487258 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.487217    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-kn69w\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.487258 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.487243    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-kn69w\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.489162 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.489144    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-kn69w\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.496819 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.496790    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85scq\" (UniqueName: \"kubernetes.io/projected/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kube-api-access-85scq\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-kn69w\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.531973 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.531922    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:21.808255 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:21.808226    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"]
Sep 11 16:55:21.810977 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:55:21.810947    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bae6bb6_6e2b_47ea_ab68_017c19a726bc.slice/crio-d9b6ffb7026f45624abd6d9a3f8c6472db87c0dd54f3f7c71e708143c7e108ff WatchSource:0}: Error finding container d9b6ffb7026f45624abd6d9a3f8c6472db87c0dd54f3f7c71e708143c7e108ff: Status 404 returned error can't find the container with id d9b6ffb7026f45624abd6d9a3f8c6472db87c0dd54f3f7c71e708143c7e108ff
Sep 11 16:55:22.614399 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:22.614361    2561 generic.go:356] "Generic (PLEG): container finished" podID="49c7e7db-9fb4-40e7-8989-6105e9d1e10a" containerID="f08d9592ede435e6ef31cf8421f91d843b92f1ab5ce4620f5dc96403c48f077d" exitCode=0
Sep 11 16:55:22.614399 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:22.614383    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q" event={"ID":"49c7e7db-9fb4-40e7-8989-6105e9d1e10a","Type":"ContainerDied","Data":"f08d9592ede435e6ef31cf8421f91d843b92f1ab5ce4620f5dc96403c48f077d"}
Sep 11 16:55:22.616282 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:22.616258    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w" event={"ID":"4bae6bb6-6e2b-47ea-ab68-017c19a726bc","Type":"ContainerStarted","Data":"ad4335302829e5215939536c30030385981d2e28b584ac8453a47874941e3c63"}
Sep 11 16:55:22.616415 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:22.616303    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w" event={"ID":"4bae6bb6-6e2b-47ea-ab68-017c19a726bc","Type":"ContainerStarted","Data":"d9b6ffb7026f45624abd6d9a3f8c6472db87c0dd54f3f7c71e708143c7e108ff"}
Sep 11 16:55:22.735429 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:22.735400    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:55:22.901252 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:22.901213    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvdfs\" (UniqueName: \"kubernetes.io/projected/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kube-api-access-cvdfs\") pod \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") "
Sep 11 16:55:22.901400 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:22.901294    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kserve-provision-location\") pod \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") "
Sep 11 16:55:22.901400 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:22.901341    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-proxy-tls\") pod \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") "
Sep 11 16:55:22.901400 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:22.901382    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-cabundle-cert\") pod \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") "
Sep 11 16:55:22.901529 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:22.901422    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kube-rbac-proxy-sar-config\") pod \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\" (UID: \"49c7e7db-9fb4-40e7-8989-6105e9d1e10a\") "
Sep 11 16:55:22.901679 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:22.901651    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "49c7e7db-9fb4-40e7-8989-6105e9d1e10a" (UID: "49c7e7db-9fb4-40e7-8989-6105e9d1e10a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:55:22.901800 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:22.901779    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "49c7e7db-9fb4-40e7-8989-6105e9d1e10a" (UID: "49c7e7db-9fb4-40e7-8989-6105e9d1e10a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:55:22.901869 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:22.901843    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "49c7e7db-9fb4-40e7-8989-6105e9d1e10a" (UID: "49c7e7db-9fb4-40e7-8989-6105e9d1e10a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:55:22.903473 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:22.903451    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kube-api-access-cvdfs" (OuterVolumeSpecName: "kube-api-access-cvdfs") pod "49c7e7db-9fb4-40e7-8989-6105e9d1e10a" (UID: "49c7e7db-9fb4-40e7-8989-6105e9d1e10a"). InnerVolumeSpecName "kube-api-access-cvdfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:55:22.903539 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:22.903509    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "49c7e7db-9fb4-40e7-8989-6105e9d1e10a" (UID: "49c7e7db-9fb4-40e7-8989-6105e9d1e10a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:55:23.002141 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:23.002105    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvdfs\" (UniqueName: \"kubernetes.io/projected/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kube-api-access-cvdfs\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:55:23.002141 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:23.002136    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:55:23.002141 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:23.002145    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:55:23.002388 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:23.002159    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:55:23.002388 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:23.002169    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49c7e7db-9fb4-40e7-8989-6105e9d1e10a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:55:23.623117 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:23.623031    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"
Sep 11 16:55:23.623117 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:23.623039    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q" event={"ID":"49c7e7db-9fb4-40e7-8989-6105e9d1e10a","Type":"ContainerDied","Data":"a7108b89b333266ebfe8b7890bd5760b70b3a019a78f78f1dda281e63a9067a9"}
Sep 11 16:55:23.623694 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:23.623133    2561 scope.go:117] "RemoveContainer" containerID="b6442749c019ad5a4368613fae9e903683bf9053263e07f32a16f3ef47583289"
Sep 11 16:55:23.632090 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:23.632071    2561 scope.go:117] "RemoveContainer" containerID="f08d9592ede435e6ef31cf8421f91d843b92f1ab5ce4620f5dc96403c48f077d"
Sep 11 16:55:23.639988 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:23.639959    2561 scope.go:117] "RemoveContainer" containerID="683f49a4792e1694ee1054e343e456bbead6f76b635d19d939c744372c7cd5d9"
Sep 11 16:55:23.645733 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:23.645708    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"]
Sep 11 16:55:23.651422 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:23.651397    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zdm2q"]
Sep 11 16:55:25.320955 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:25.320918    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49c7e7db-9fb4-40e7-8989-6105e9d1e10a" path="/var/lib/kubelet/pods/49c7e7db-9fb4-40e7-8989-6105e9d1e10a/volumes"
Sep 11 16:55:26.642118 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:26.642083    2561 generic.go:356] "Generic (PLEG): container finished" podID="4bae6bb6-6e2b-47ea-ab68-017c19a726bc" containerID="ad4335302829e5215939536c30030385981d2e28b584ac8453a47874941e3c63" exitCode=0
Sep 11 16:55:26.642542 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:26.642125    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w" event={"ID":"4bae6bb6-6e2b-47ea-ab68-017c19a726bc","Type":"ContainerDied","Data":"ad4335302829e5215939536c30030385981d2e28b584ac8453a47874941e3c63"}
Sep 11 16:55:27.650290 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:27.650206    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w" event={"ID":"4bae6bb6-6e2b-47ea-ab68-017c19a726bc","Type":"ContainerStarted","Data":"8203a109908e26b655d1c173822ecb1279c96e705cc765797be66c2dc8a27341"}
Sep 11 16:55:27.650290 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:27.650281    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w" event={"ID":"4bae6bb6-6e2b-47ea-ab68-017c19a726bc","Type":"ContainerStarted","Data":"6bfa6c2815f44b4ab029ec33040ae7fc025bb0e374735e997237d6173290db46"}
Sep 11 16:55:27.650696 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:27.650478    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:27.675198 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:27.675122    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w" podStartSLOduration=2.593813463 podStartE2EDuration="6.67510359s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.081290127s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:55:27.673772674 +0000 UTC m=+2333.013866720" watchObservedRunningTime="2026-09-11 16:55:27.67510359 +0000 UTC m=+2333.015197635"
Sep 11 16:55:28.655544 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:28.655507    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:29.765381 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:29.765338    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"]
Sep 11 16:55:29.765854 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:29.765634    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kserve-container" containerID="cri-o://bafcf6976942367b487a0658b933668d0b2ac25ab4a62475365afb5858343ad4" gracePeriod=30
Sep 11 16:55:29.765854 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:29.765685    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kube-rbac-proxy" containerID="cri-o://b330a131ae29db8c9b14ca29d8c0ae5876b6d4068c9686a2569bb100e6577aaa" gracePeriod=30
Sep 11 16:55:30.671344 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:30.671309    2561 generic.go:356] "Generic (PLEG): container finished" podID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerID="b330a131ae29db8c9b14ca29d8c0ae5876b6d4068c9686a2569bb100e6577aaa" exitCode=2
Sep 11 16:55:30.671504 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:30.671352    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" event={"ID":"44f0d988-c82b-4f24-9556-ff1ecd9c03de","Type":"ContainerDied","Data":"b330a131ae29db8c9b14ca29d8c0ae5876b6d4068c9686a2569bb100e6577aaa"}
Sep 11 16:55:31.614357 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:31.614312    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 11 16:55:31.618719 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:31.618693    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 11 16:55:34.610406 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.610378    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:55:34.665652 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.665618    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:55:34.695226 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.695132    2561 generic.go:356] "Generic (PLEG): container finished" podID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerID="bafcf6976942367b487a0658b933668d0b2ac25ab4a62475365afb5858343ad4" exitCode=0
Sep 11 16:55:34.695226 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.695217    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"
Sep 11 16:55:34.695407 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.695208    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" event={"ID":"44f0d988-c82b-4f24-9556-ff1ecd9c03de","Type":"ContainerDied","Data":"bafcf6976942367b487a0658b933668d0b2ac25ab4a62475365afb5858343ad4"}
Sep 11 16:55:34.695407 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.695345    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk" event={"ID":"44f0d988-c82b-4f24-9556-ff1ecd9c03de","Type":"ContainerDied","Data":"a2fedd6bed5959bd2a97202eebeec928cf33d9d4a6f1ebd96b3020b0b25adbce"}
Sep 11 16:55:34.695407 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.695375    2561 scope.go:117] "RemoveContainer" containerID="b330a131ae29db8c9b14ca29d8c0ae5876b6d4068c9686a2569bb100e6577aaa"
Sep 11 16:55:34.703751 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.703732    2561 scope.go:117] "RemoveContainer" containerID="bafcf6976942367b487a0658b933668d0b2ac25ab4a62475365afb5858343ad4"
Sep 11 16:55:34.709966 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.709940    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f0d988-c82b-4f24-9556-ff1ecd9c03de-proxy-tls\") pod \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") "
Sep 11 16:55:34.710076 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.710003    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44f0d988-c82b-4f24-9556-ff1ecd9c03de-cabundle-cert\") pod \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") "
Sep 11 16:55:34.710143 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.710132    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kserve-provision-location\") pod \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") "
Sep 11 16:55:34.710232 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.710206    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq9rv\" (UniqueName: \"kubernetes.io/projected/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kube-api-access-dq9rv\") pod \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") "
Sep 11 16:55:34.710322 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.710284    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kube-rbac-proxy-sar-config\") pod \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\" (UID: \"44f0d988-c82b-4f24-9556-ff1ecd9c03de\") "
Sep 11 16:55:34.710393 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.710379    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f0d988-c82b-4f24-9556-ff1ecd9c03de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "44f0d988-c82b-4f24-9556-ff1ecd9c03de" (UID: "44f0d988-c82b-4f24-9556-ff1ecd9c03de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:55:34.710463 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.710436    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "44f0d988-c82b-4f24-9556-ff1ecd9c03de" (UID: "44f0d988-c82b-4f24-9556-ff1ecd9c03de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:55:34.710711 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.710673    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "44f0d988-c82b-4f24-9556-ff1ecd9c03de" (UID: "44f0d988-c82b-4f24-9556-ff1ecd9c03de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:55:34.710854 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.710787    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44f0d988-c82b-4f24-9556-ff1ecd9c03de-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:55:34.710925 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.710855    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:55:34.712144 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.712125    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f0d988-c82b-4f24-9556-ff1ecd9c03de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44f0d988-c82b-4f24-9556-ff1ecd9c03de" (UID: "44f0d988-c82b-4f24-9556-ff1ecd9c03de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:55:34.712719 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.712701    2561 scope.go:117] "RemoveContainer" containerID="595dcec535b681b942875d01a4ae221dffe700cf589ba14b6ad9bc0459ceb1d9"
Sep 11 16:55:34.712863 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.712835    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kube-api-access-dq9rv" (OuterVolumeSpecName: "kube-api-access-dq9rv") pod "44f0d988-c82b-4f24-9556-ff1ecd9c03de" (UID: "44f0d988-c82b-4f24-9556-ff1ecd9c03de"). InnerVolumeSpecName "kube-api-access-dq9rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:55:34.728316 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.728291    2561 scope.go:117] "RemoveContainer" containerID="b330a131ae29db8c9b14ca29d8c0ae5876b6d4068c9686a2569bb100e6577aaa"
Sep 11 16:55:34.728663 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:55:34.728643    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b330a131ae29db8c9b14ca29d8c0ae5876b6d4068c9686a2569bb100e6577aaa\": container with ID starting with b330a131ae29db8c9b14ca29d8c0ae5876b6d4068c9686a2569bb100e6577aaa not found: ID does not exist" containerID="b330a131ae29db8c9b14ca29d8c0ae5876b6d4068c9686a2569bb100e6577aaa"
Sep 11 16:55:34.728743 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.728671    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b330a131ae29db8c9b14ca29d8c0ae5876b6d4068c9686a2569bb100e6577aaa"} err="failed to get container status \"b330a131ae29db8c9b14ca29d8c0ae5876b6d4068c9686a2569bb100e6577aaa\": rpc error: code = NotFound desc = could not find container \"b330a131ae29db8c9b14ca29d8c0ae5876b6d4068c9686a2569bb100e6577aaa\": container with ID starting with b330a131ae29db8c9b14ca29d8c0ae5876b6d4068c9686a2569bb100e6577aaa not found: ID does not exist"
Sep 11 16:55:34.728743 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.728688    2561 scope.go:117] "RemoveContainer" containerID="bafcf6976942367b487a0658b933668d0b2ac25ab4a62475365afb5858343ad4"
Sep 11 16:55:34.728951 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:55:34.728930    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bafcf6976942367b487a0658b933668d0b2ac25ab4a62475365afb5858343ad4\": container with ID starting with bafcf6976942367b487a0658b933668d0b2ac25ab4a62475365afb5858343ad4 not found: ID does not exist" containerID="bafcf6976942367b487a0658b933668d0b2ac25ab4a62475365afb5858343ad4"
Sep 11 16:55:34.729001 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.728962    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafcf6976942367b487a0658b933668d0b2ac25ab4a62475365afb5858343ad4"} err="failed to get container status \"bafcf6976942367b487a0658b933668d0b2ac25ab4a62475365afb5858343ad4\": rpc error: code = NotFound desc = could not find container \"bafcf6976942367b487a0658b933668d0b2ac25ab4a62475365afb5858343ad4\": container with ID starting with bafcf6976942367b487a0658b933668d0b2ac25ab4a62475365afb5858343ad4 not found: ID does not exist"
Sep 11 16:55:34.729001 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.728980    2561 scope.go:117] "RemoveContainer" containerID="595dcec535b681b942875d01a4ae221dffe700cf589ba14b6ad9bc0459ceb1d9"
Sep 11 16:55:34.729346 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:55:34.729324    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"595dcec535b681b942875d01a4ae221dffe700cf589ba14b6ad9bc0459ceb1d9\": container with ID starting with 595dcec535b681b942875d01a4ae221dffe700cf589ba14b6ad9bc0459ceb1d9 not found: ID does not exist" containerID="595dcec535b681b942875d01a4ae221dffe700cf589ba14b6ad9bc0459ceb1d9"
Sep 11 16:55:34.729417 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.729348    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"595dcec535b681b942875d01a4ae221dffe700cf589ba14b6ad9bc0459ceb1d9"} err="failed to get container status \"595dcec535b681b942875d01a4ae221dffe700cf589ba14b6ad9bc0459ceb1d9\": rpc error: code = NotFound desc = could not find container \"595dcec535b681b942875d01a4ae221dffe700cf589ba14b6ad9bc0459ceb1d9\": container with ID starting with 595dcec535b681b942875d01a4ae221dffe700cf589ba14b6ad9bc0459ceb1d9 not found: ID does not exist"
Sep 11 16:55:34.811652 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.811609    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq9rv\" (UniqueName: \"kubernetes.io/projected/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kube-api-access-dq9rv\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:55:34.811652 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.811644    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44f0d988-c82b-4f24-9556-ff1ecd9c03de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:55:34.811652 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:34.811656    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44f0d988-c82b-4f24-9556-ff1ecd9c03de-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:55:35.033552 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.033515    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"]
Sep 11 16:55:35.043250 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.043204    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-8wwpk"]
Sep 11 16:55:35.297995 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.297908    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"]
Sep 11 16:55:35.298305 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.298291    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49c7e7db-9fb4-40e7-8989-6105e9d1e10a" containerName="kserve-container"
Sep 11 16:55:35.298358 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.298306    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="49c7e7db-9fb4-40e7-8989-6105e9d1e10a" containerName="kserve-container"
Sep 11 16:55:35.298358 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.298328    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="storage-initializer"
Sep 11 16:55:35.298358 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.298334    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="storage-initializer"
Sep 11 16:55:35.298358 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.298344    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kube-rbac-proxy"
Sep 11 16:55:35.298358 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.298349    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kube-rbac-proxy"
Sep 11 16:55:35.298358 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.298356    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kserve-container"
Sep 11 16:55:35.298358 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.298361    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kserve-container"
Sep 11 16:55:35.298561 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.298367    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49c7e7db-9fb4-40e7-8989-6105e9d1e10a" containerName="kube-rbac-proxy"
Sep 11 16:55:35.298561 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.298373    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="49c7e7db-9fb4-40e7-8989-6105e9d1e10a" containerName="kube-rbac-proxy"
Sep 11 16:55:35.298561 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.298384    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49c7e7db-9fb4-40e7-8989-6105e9d1e10a" containerName="storage-initializer"
Sep 11 16:55:35.298561 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.298389    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="49c7e7db-9fb4-40e7-8989-6105e9d1e10a" containerName="storage-initializer"
Sep 11 16:55:35.298561 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.298444    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="49c7e7db-9fb4-40e7-8989-6105e9d1e10a" containerName="kube-rbac-proxy"
Sep 11 16:55:35.298561 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.298454    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="49c7e7db-9fb4-40e7-8989-6105e9d1e10a" containerName="kserve-container"
Sep 11 16:55:35.298561 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.298460    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kube-rbac-proxy"
Sep 11 16:55:35.298561 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.298466    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" containerName="kserve-container"
Sep 11 16:55:35.303265 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.303242    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:35.305773 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.305744    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 11 16:55:35.305773 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.305748    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 11 16:55:35.323477 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.323441    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44f0d988-c82b-4f24-9556-ff1ecd9c03de" path="/var/lib/kubelet/pods/44f0d988-c82b-4f24-9556-ff1ecd9c03de/volumes"
Sep 11 16:55:35.323975 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.323954    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"]
Sep 11 16:55:35.417078 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.417033    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-mth9z\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:35.417078 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.417078    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-mth9z\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:35.417324 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.417146    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-mth9z\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:35.417324 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.417243    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27frn\" (UniqueName: \"kubernetes.io/projected/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kube-api-access-27frn\") pod \"isvc-canary-stop-predictor-5cdcf6545-mth9z\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:35.417324 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.417261    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-mth9z\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:35.518254 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.518202    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-mth9z\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:35.518435 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.518296    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27frn\" (UniqueName: \"kubernetes.io/projected/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kube-api-access-27frn\") pod \"isvc-canary-stop-predictor-5cdcf6545-mth9z\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:35.518435 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.518326    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-mth9z\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:35.518435 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.518379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-mth9z\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:35.518435 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.518405    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-mth9z\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:35.518614 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:55:35.518531    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 11 16:55:35.518614 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.518570    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-mth9z\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:35.518614 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:55:35.518603    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-proxy-tls podName:b73d107d-d7ea-4b6e-86b5-43bc3ac4194a nodeName:}" failed. No retries permitted until 2026-09-11 16:55:36.018581034 +0000 UTC m=+2341.358675063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-proxy-tls") pod "isvc-canary-stop-predictor-5cdcf6545-mth9z" (UID: "b73d107d-d7ea-4b6e-86b5-43bc3ac4194a") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 11 16:55:35.519020 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.518999    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-mth9z\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:35.519085 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.519052    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-mth9z\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:35.530081 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:35.530049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27frn\" (UniqueName: \"kubernetes.io/projected/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kube-api-access-27frn\") pod \"isvc-canary-stop-predictor-5cdcf6545-mth9z\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:36.022977 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:36.022938    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-mth9z\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:36.025411 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:36.025391    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-mth9z\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:36.213166 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:36.213127    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:36.491977 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:36.491951    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"]
Sep 11 16:55:36.494698 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:55:36.494664    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73d107d_d7ea_4b6e_86b5_43bc3ac4194a.slice/crio-ff975ad23d0ebe70746f73c799a7e00ecb6ad727a72cfe7819969b624c4c75a3 WatchSource:0}: Error finding container ff975ad23d0ebe70746f73c799a7e00ecb6ad727a72cfe7819969b624c4c75a3: Status 404 returned error can't find the container with id ff975ad23d0ebe70746f73c799a7e00ecb6ad727a72cfe7819969b624c4c75a3
Sep 11 16:55:36.711069 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:36.711031    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" event={"ID":"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a","Type":"ContainerStarted","Data":"56a1dcc705b5d33493b3dddd56acc3aacc46eca7195778a01baa2038aa49d862"}
Sep 11 16:55:36.711255 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:36.711093    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" event={"ID":"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a","Type":"ContainerStarted","Data":"ff975ad23d0ebe70746f73c799a7e00ecb6ad727a72cfe7819969b624c4c75a3"}
Sep 11 16:55:40.731571 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:40.731535    2561 generic.go:356] "Generic (PLEG): container finished" podID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerID="56a1dcc705b5d33493b3dddd56acc3aacc46eca7195778a01baa2038aa49d862" exitCode=0
Sep 11 16:55:40.731947 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:40.731609    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" event={"ID":"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a","Type":"ContainerDied","Data":"56a1dcc705b5d33493b3dddd56acc3aacc46eca7195778a01baa2038aa49d862"}
Sep 11 16:55:41.738452 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:41.738416    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" event={"ID":"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a","Type":"ContainerStarted","Data":"69137d881d4c20938171c5a45d3903d5743fd172efd77bbc7e36774ed2882686"}
Sep 11 16:55:41.738941 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:41.738490    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" event={"ID":"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a","Type":"ContainerStarted","Data":"663396d3950af6014042b871ac3cecf893bbe36d48d7b4917657f62ba14ad26a"}
Sep 11 16:55:41.738941 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:41.738712    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:41.763443 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:41.763387    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" podStartSLOduration=2.969761671 podStartE2EDuration="6.763370018s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.793608347s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:55:41.761565827 +0000 UTC m=+2347.101659872" watchObservedRunningTime="2026-09-11 16:55:41.763370018 +0000 UTC m=+2347.103464063"
Sep 11 16:55:42.745223 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:42.745190    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:42.746114 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:42.746082    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 11 16:55:43.749785 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:43.749746    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 11 16:55:48.754800 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:48.754759    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:55:48.755507 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:48.755477    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 11 16:55:58.755504 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:55:58.755460    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 11 16:56:04.741448 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:04.741403    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:56:08.755771 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:08.755731    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 11 16:56:11.225772 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:11.225735    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"]
Sep 11 16:56:11.226255 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:11.225989    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w" podUID="4bae6bb6-6e2b-47ea-ab68-017c19a726bc" containerName="kube-rbac-proxy" containerID="cri-o://8203a109908e26b655d1c173822ecb1279c96e705cc765797be66c2dc8a27341" gracePeriod=30
Sep 11 16:56:11.226255 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:11.225977    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w" podUID="4bae6bb6-6e2b-47ea-ab68-017c19a726bc" containerName="kserve-container" containerID="cri-o://6bfa6c2815f44b4ab029ec33040ae7fc025bb0e374735e997237d6173290db46" gracePeriod=30
Sep 11 16:56:11.897376 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:11.897336    2561 generic.go:356] "Generic (PLEG): container finished" podID="4bae6bb6-6e2b-47ea-ab68-017c19a726bc" containerID="8203a109908e26b655d1c173822ecb1279c96e705cc765797be66c2dc8a27341" exitCode=2
Sep 11 16:56:11.897543 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:11.897409    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w" event={"ID":"4bae6bb6-6e2b-47ea-ab68-017c19a726bc","Type":"ContainerDied","Data":"8203a109908e26b655d1c173822ecb1279c96e705cc765797be66c2dc8a27341"}
Sep 11 16:56:14.661840 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:14.661789    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w" podUID="4bae6bb6-6e2b-47ea-ab68-017c19a726bc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 11 16:56:14.666402 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:14.666370    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w" podUID="4bae6bb6-6e2b-47ea-ab68-017c19a726bc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 11 16:56:17.930678 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:17.930635    2561 generic.go:356] "Generic (PLEG): container finished" podID="4bae6bb6-6e2b-47ea-ab68-017c19a726bc" containerID="6bfa6c2815f44b4ab029ec33040ae7fc025bb0e374735e997237d6173290db46" exitCode=0
Sep 11 16:56:17.931073 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:17.930699    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w" event={"ID":"4bae6bb6-6e2b-47ea-ab68-017c19a726bc","Type":"ContainerDied","Data":"6bfa6c2815f44b4ab029ec33040ae7fc025bb0e374735e997237d6173290db46"}
Sep 11 16:56:17.975911 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:17.975883    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:56:18.003733 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.003652    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kserve-provision-location\") pod \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") "
Sep 11 16:56:18.003891 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.003738    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-cabundle-cert\") pod \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") "
Sep 11 16:56:18.003891 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.003815    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kube-rbac-proxy-sar-config\") pod \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") "
Sep 11 16:56:18.003891 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.003875    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85scq\" (UniqueName: \"kubernetes.io/projected/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kube-api-access-85scq\") pod \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") "
Sep 11 16:56:18.004050 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.003906    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-proxy-tls\") pod \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\" (UID: \"4bae6bb6-6e2b-47ea-ab68-017c19a726bc\") "
Sep 11 16:56:18.004050 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.004008    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4bae6bb6-6e2b-47ea-ab68-017c19a726bc" (UID: "4bae6bb6-6e2b-47ea-ab68-017c19a726bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:56:18.004155 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.004092    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4bae6bb6-6e2b-47ea-ab68-017c19a726bc" (UID: "4bae6bb6-6e2b-47ea-ab68-017c19a726bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:56:18.004253 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.004230    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4bae6bb6-6e2b-47ea-ab68-017c19a726bc" (UID: "4bae6bb6-6e2b-47ea-ab68-017c19a726bc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:56:18.004312 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.004267    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:56:18.004312 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.004288    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:56:18.006204 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.006157    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4bae6bb6-6e2b-47ea-ab68-017c19a726bc" (UID: "4bae6bb6-6e2b-47ea-ab68-017c19a726bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:56:18.006204 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.006193    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kube-api-access-85scq" (OuterVolumeSpecName: "kube-api-access-85scq") pod "4bae6bb6-6e2b-47ea-ab68-017c19a726bc" (UID: "4bae6bb6-6e2b-47ea-ab68-017c19a726bc"). InnerVolumeSpecName "kube-api-access-85scq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:56:18.105050 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.105016    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85scq\" (UniqueName: \"kubernetes.io/projected/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kube-api-access-85scq\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:56:18.105050 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.105044    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:56:18.105050 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.105054    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bae6bb6-6e2b-47ea-ab68-017c19a726bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:56:18.756212 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.756156    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 11 16:56:18.937305 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.937274    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"
Sep 11 16:56:18.937730 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.937300    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w" event={"ID":"4bae6bb6-6e2b-47ea-ab68-017c19a726bc","Type":"ContainerDied","Data":"d9b6ffb7026f45624abd6d9a3f8c6472db87c0dd54f3f7c71e708143c7e108ff"}
Sep 11 16:56:18.937730 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.937381    2561 scope.go:117] "RemoveContainer" containerID="8203a109908e26b655d1c173822ecb1279c96e705cc765797be66c2dc8a27341"
Sep 11 16:56:18.947875 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.947854    2561 scope.go:117] "RemoveContainer" containerID="6bfa6c2815f44b4ab029ec33040ae7fc025bb0e374735e997237d6173290db46"
Sep 11 16:56:18.956107 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.956084    2561 scope.go:117] "RemoveContainer" containerID="ad4335302829e5215939536c30030385981d2e28b584ac8453a47874941e3c63"
Sep 11 16:56:18.962397 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.962366    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"]
Sep 11 16:56:18.967525 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:18.967498    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-kn69w"]
Sep 11 16:56:19.321523 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:19.321490    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bae6bb6-6e2b-47ea-ab68-017c19a726bc" path="/var/lib/kubelet/pods/4bae6bb6-6e2b-47ea-ab68-017c19a726bc/volumes"
Sep 11 16:56:28.756191 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:28.756127    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 11 16:56:35.342873 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:35.342843    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:56:35.459302 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:35.459269    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 16:56:38.756331 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:38.756290    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 11 16:56:48.756090 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:48.756042    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 11 16:56:49.316744 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:49.316701    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 11 16:56:59.321095 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:56:59.321018    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:57:10.098663 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:10.098626    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"]
Sep 11 16:57:10.099197 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:10.098865    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kserve-container" containerID="cri-o://663396d3950af6014042b871ac3cecf893bbe36d48d7b4917657f62ba14ad26a" gracePeriod=30
Sep 11 16:57:10.099197 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:10.098922    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kube-rbac-proxy" containerID="cri-o://69137d881d4c20938171c5a45d3903d5743fd172efd77bbc7e36774ed2882686" gracePeriod=30
Sep 11 16:57:11.184517 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:11.184484    2561 generic.go:356] "Generic (PLEG): container finished" podID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerID="69137d881d4c20938171c5a45d3903d5743fd172efd77bbc7e36774ed2882686" exitCode=2
Sep 11 16:57:11.184886 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:11.184558    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" event={"ID":"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a","Type":"ContainerDied","Data":"69137d881d4c20938171c5a45d3903d5743fd172efd77bbc7e36774ed2882686"}
Sep 11 16:57:13.750530 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:13.750484    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 11 16:57:14.951526 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:14.951498    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:57:14.999892 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:14.999804    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27frn\" (UniqueName: \"kubernetes.io/projected/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kube-api-access-27frn\") pod \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") "
Sep 11 16:57:14.999892 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:14.999862    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kserve-provision-location\") pod \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") "
Sep 11 16:57:14.999892 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:14.999893    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kube-rbac-proxy-sar-config\") pod \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") "
Sep 11 16:57:15.000202 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:14.999935    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-proxy-tls\") pod \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") "
Sep 11 16:57:15.000202 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:14.999971    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-cabundle-cert\") pod \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\" (UID: \"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a\") "
Sep 11 16:57:15.000366 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.000272    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" (UID: "b73d107d-d7ea-4b6e-86b5-43bc3ac4194a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:57:15.000408 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.000385    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" (UID: "b73d107d-d7ea-4b6e-86b5-43bc3ac4194a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:57:15.000462 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.000407    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" (UID: "b73d107d-d7ea-4b6e-86b5-43bc3ac4194a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:57:15.002138 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.002096    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kube-api-access-27frn" (OuterVolumeSpecName: "kube-api-access-27frn") pod "b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" (UID: "b73d107d-d7ea-4b6e-86b5-43bc3ac4194a"). InnerVolumeSpecName "kube-api-access-27frn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:57:15.002138 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.002095    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" (UID: "b73d107d-d7ea-4b6e-86b5-43bc3ac4194a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:57:15.100874 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.100842    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:57:15.100874 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.100870    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:57:15.100874 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.100881    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:57:15.101110 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.100890    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:57:15.101110 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.100898    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27frn\" (UniqueName: \"kubernetes.io/projected/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a-kube-api-access-27frn\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:57:15.205150 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.205110    2561 generic.go:356] "Generic (PLEG): container finished" podID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerID="663396d3950af6014042b871ac3cecf893bbe36d48d7b4917657f62ba14ad26a" exitCode=0
Sep 11 16:57:15.205333 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.205162    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" event={"ID":"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a","Type":"ContainerDied","Data":"663396d3950af6014042b871ac3cecf893bbe36d48d7b4917657f62ba14ad26a"}
Sep 11 16:57:15.205333 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.205170    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"
Sep 11 16:57:15.205333 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.205231    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z" event={"ID":"b73d107d-d7ea-4b6e-86b5-43bc3ac4194a","Type":"ContainerDied","Data":"ff975ad23d0ebe70746f73c799a7e00ecb6ad727a72cfe7819969b624c4c75a3"}
Sep 11 16:57:15.205333 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.205259    2561 scope.go:117] "RemoveContainer" containerID="69137d881d4c20938171c5a45d3903d5743fd172efd77bbc7e36774ed2882686"
Sep 11 16:57:15.214942 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.214920    2561 scope.go:117] "RemoveContainer" containerID="663396d3950af6014042b871ac3cecf893bbe36d48d7b4917657f62ba14ad26a"
Sep 11 16:57:15.222674 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.222655    2561 scope.go:117] "RemoveContainer" containerID="56a1dcc705b5d33493b3dddd56acc3aacc46eca7195778a01baa2038aa49d862"
Sep 11 16:57:15.229768 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.229743    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"]
Sep 11 16:57:15.230783 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.230768    2561 scope.go:117] "RemoveContainer" containerID="69137d881d4c20938171c5a45d3903d5743fd172efd77bbc7e36774ed2882686"
Sep 11 16:57:15.231061 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:57:15.231041    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69137d881d4c20938171c5a45d3903d5743fd172efd77bbc7e36774ed2882686\": container with ID starting with 69137d881d4c20938171c5a45d3903d5743fd172efd77bbc7e36774ed2882686 not found: ID does not exist" containerID="69137d881d4c20938171c5a45d3903d5743fd172efd77bbc7e36774ed2882686"
Sep 11 16:57:15.231138 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.231075    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69137d881d4c20938171c5a45d3903d5743fd172efd77bbc7e36774ed2882686"} err="failed to get container status \"69137d881d4c20938171c5a45d3903d5743fd172efd77bbc7e36774ed2882686\": rpc error: code = NotFound desc = could not find container \"69137d881d4c20938171c5a45d3903d5743fd172efd77bbc7e36774ed2882686\": container with ID starting with 69137d881d4c20938171c5a45d3903d5743fd172efd77bbc7e36774ed2882686 not found: ID does not exist"
Sep 11 16:57:15.231138 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.231102    2561 scope.go:117] "RemoveContainer" containerID="663396d3950af6014042b871ac3cecf893bbe36d48d7b4917657f62ba14ad26a"
Sep 11 16:57:15.231386 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:57:15.231369    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663396d3950af6014042b871ac3cecf893bbe36d48d7b4917657f62ba14ad26a\": container with ID starting with 663396d3950af6014042b871ac3cecf893bbe36d48d7b4917657f62ba14ad26a not found: ID does not exist" containerID="663396d3950af6014042b871ac3cecf893bbe36d48d7b4917657f62ba14ad26a"
Sep 11 16:57:15.231439 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.231391    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663396d3950af6014042b871ac3cecf893bbe36d48d7b4917657f62ba14ad26a"} err="failed to get container status \"663396d3950af6014042b871ac3cecf893bbe36d48d7b4917657f62ba14ad26a\": rpc error: code = NotFound desc = could not find container \"663396d3950af6014042b871ac3cecf893bbe36d48d7b4917657f62ba14ad26a\": container with ID starting with 663396d3950af6014042b871ac3cecf893bbe36d48d7b4917657f62ba14ad26a not found: ID does not exist"
Sep 11 16:57:15.231439 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.231405    2561 scope.go:117] "RemoveContainer" containerID="56a1dcc705b5d33493b3dddd56acc3aacc46eca7195778a01baa2038aa49d862"
Sep 11 16:57:15.231673 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:57:15.231656    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56a1dcc705b5d33493b3dddd56acc3aacc46eca7195778a01baa2038aa49d862\": container with ID starting with 56a1dcc705b5d33493b3dddd56acc3aacc46eca7195778a01baa2038aa49d862 not found: ID does not exist" containerID="56a1dcc705b5d33493b3dddd56acc3aacc46eca7195778a01baa2038aa49d862"
Sep 11 16:57:15.231726 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.231679    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56a1dcc705b5d33493b3dddd56acc3aacc46eca7195778a01baa2038aa49d862"} err="failed to get container status \"56a1dcc705b5d33493b3dddd56acc3aacc46eca7195778a01baa2038aa49d862\": rpc error: code = NotFound desc = could not find container \"56a1dcc705b5d33493b3dddd56acc3aacc46eca7195778a01baa2038aa49d862\": container with ID starting with 56a1dcc705b5d33493b3dddd56acc3aacc46eca7195778a01baa2038aa49d862 not found: ID does not exist"
Sep 11 16:57:15.241337 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.241312    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-mth9z"]
Sep 11 16:57:15.322718 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:57:15.322633    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" path="/var/lib/kubelet/pods/b73d107d-d7ea-4b6e-86b5-43bc3ac4194a/volumes"
Sep 11 16:58:01.925810 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.925768    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"]
Sep 11 16:58:01.926245 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.926123    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kserve-container"
Sep 11 16:58:01.926245 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.926134    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kserve-container"
Sep 11 16:58:01.926245 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.926142    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bae6bb6-6e2b-47ea-ab68-017c19a726bc" containerName="kube-rbac-proxy"
Sep 11 16:58:01.926245 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.926148    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bae6bb6-6e2b-47ea-ab68-017c19a726bc" containerName="kube-rbac-proxy"
Sep 11 16:58:01.926245 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.926167    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="storage-initializer"
Sep 11 16:58:01.926245 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.926187    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="storage-initializer"
Sep 11 16:58:01.926245 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.926204    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kube-rbac-proxy"
Sep 11 16:58:01.926245 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.926212    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kube-rbac-proxy"
Sep 11 16:58:01.926245 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.926226    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bae6bb6-6e2b-47ea-ab68-017c19a726bc" containerName="storage-initializer"
Sep 11 16:58:01.926245 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.926234    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bae6bb6-6e2b-47ea-ab68-017c19a726bc" containerName="storage-initializer"
Sep 11 16:58:01.926245 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.926242    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bae6bb6-6e2b-47ea-ab68-017c19a726bc" containerName="kserve-container"
Sep 11 16:58:01.926245 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.926247    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bae6bb6-6e2b-47ea-ab68-017c19a726bc" containerName="kserve-container"
Sep 11 16:58:01.926627 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.926304    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kube-rbac-proxy"
Sep 11 16:58:01.926627 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.926311    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bae6bb6-6e2b-47ea-ab68-017c19a726bc" containerName="kserve-container"
Sep 11 16:58:01.926627 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.926317    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bae6bb6-6e2b-47ea-ab68-017c19a726bc" containerName="kube-rbac-proxy"
Sep 11 16:58:01.926627 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.926323    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b73d107d-d7ea-4b6e-86b5-43bc3ac4194a" containerName="kserve-container"
Sep 11 16:58:01.929544 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.929523    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:01.933364 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.933339    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:58:01.933630 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.933612    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 11 16:58:01.933738 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.933718    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 16:58:01.936622 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.936594    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:58:01.936741 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.936669    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:58:01.937402 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.937385    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 11 16:58:01.940246 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.940226    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:58:01.949553 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:01.949525    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"]
Sep 11 16:58:02.006423 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.006382    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be427076-3fa7-46ee-89d1-c83bf42103f1-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:02.006594 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.006436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be427076-3fa7-46ee-89d1-c83bf42103f1-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:02.006594 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.006469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbk6l\" (UniqueName: \"kubernetes.io/projected/be427076-3fa7-46ee-89d1-c83bf42103f1-kube-api-access-sbk6l\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:02.006594 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.006511    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be427076-3fa7-46ee-89d1-c83bf42103f1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:02.006594 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.006543    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be427076-3fa7-46ee-89d1-c83bf42103f1-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:02.107794 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.107756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be427076-3fa7-46ee-89d1-c83bf42103f1-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:02.107969 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.107801    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbk6l\" (UniqueName: \"kubernetes.io/projected/be427076-3fa7-46ee-89d1-c83bf42103f1-kube-api-access-sbk6l\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:02.107969 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.107825    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be427076-3fa7-46ee-89d1-c83bf42103f1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:02.107969 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.107858    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be427076-3fa7-46ee-89d1-c83bf42103f1-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:02.107969 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.107893    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be427076-3fa7-46ee-89d1-c83bf42103f1-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:02.108222 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.108201    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be427076-3fa7-46ee-89d1-c83bf42103f1-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:02.108596 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.108572    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be427076-3fa7-46ee-89d1-c83bf42103f1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:02.108687 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.108578    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be427076-3fa7-46ee-89d1-c83bf42103f1-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:02.110520 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.110496    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be427076-3fa7-46ee-89d1-c83bf42103f1-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:02.119906 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.119881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbk6l\" (UniqueName: \"kubernetes.io/projected/be427076-3fa7-46ee-89d1-c83bf42103f1-kube-api-access-sbk6l\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:02.239669 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.239570    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:02.520853 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:02.520773    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"]
Sep 11 16:58:02.524047 ip-10-0-142-248 kubenswrapper[2561]: W0911 16:58:02.524019    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe427076_3fa7_46ee_89d1_c83bf42103f1.slice/crio-b56820a8a16b186a54514340e9f3011e10ee7437d82136118d939602b219b916 WatchSource:0}: Error finding container b56820a8a16b186a54514340e9f3011e10ee7437d82136118d939602b219b916: Status 404 returned error can't find the container with id b56820a8a16b186a54514340e9f3011e10ee7437d82136118d939602b219b916
Sep 11 16:58:03.426525 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:03.426490    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm" event={"ID":"be427076-3fa7-46ee-89d1-c83bf42103f1","Type":"ContainerStarted","Data":"1eec88f1ef369b6a4f44ff40b6e5b8bfeafc343a03c17199c13469c6d6b56639"}
Sep 11 16:58:03.426939 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:03.426552    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm" event={"ID":"be427076-3fa7-46ee-89d1-c83bf42103f1","Type":"ContainerStarted","Data":"b56820a8a16b186a54514340e9f3011e10ee7437d82136118d939602b219b916"}
Sep 11 16:58:07.447226 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:07.447167    2561 generic.go:356] "Generic (PLEG): container finished" podID="be427076-3fa7-46ee-89d1-c83bf42103f1" containerID="1eec88f1ef369b6a4f44ff40b6e5b8bfeafc343a03c17199c13469c6d6b56639" exitCode=0
Sep 11 16:58:07.447625 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:07.447241    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm" event={"ID":"be427076-3fa7-46ee-89d1-c83bf42103f1","Type":"ContainerDied","Data":"1eec88f1ef369b6a4f44ff40b6e5b8bfeafc343a03c17199c13469c6d6b56639"}
Sep 11 16:58:08.454401 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:08.454365    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm" event={"ID":"be427076-3fa7-46ee-89d1-c83bf42103f1","Type":"ContainerStarted","Data":"b18b97ae703cb75d188d16ad136702ed0938be75545ecccab7ba33be986544b4"}
Sep 11 16:58:08.454776 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:08.454430    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm" event={"ID":"be427076-3fa7-46ee-89d1-c83bf42103f1","Type":"ContainerStarted","Data":"72c05d4b3f9fc816c0e811aebc682bcb643633481bb5ef09077982b0b17c3507"}
Sep 11 16:58:08.454776 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:08.454649    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:08.475329 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:08.475279    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm" podStartSLOduration=3.432112138 podStartE2EDuration="7.47526364s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.043151502s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:58:08.474507623 +0000 UTC m=+2493.814601694" watchObservedRunningTime="2026-09-11 16:58:08.47526364 +0000 UTC m=+2493.815357686"
Sep 11 16:58:09.460192 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:09.460150    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:15.472376 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:15.472343    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:58:45.541650 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:45.541607    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm" podUID="be427076-3fa7-46ee-89d1-c83bf42103f1" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 11 16:58:55.475149 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:58:55.475120    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:59:01.811831 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:01.811794    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"]
Sep 11 16:59:01.812249 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:01.812033    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm" podUID="be427076-3fa7-46ee-89d1-c83bf42103f1" containerName="kserve-container" containerID="cri-o://72c05d4b3f9fc816c0e811aebc682bcb643633481bb5ef09077982b0b17c3507" gracePeriod=30
Sep 11 16:59:01.812249 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:01.812063    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm" podUID="be427076-3fa7-46ee-89d1-c83bf42103f1" containerName="kube-rbac-proxy" containerID="cri-o://b18b97ae703cb75d188d16ad136702ed0938be75545ecccab7ba33be986544b4" gracePeriod=30
Sep 11 16:59:02.717495 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:02.717465    2561 generic.go:356] "Generic (PLEG): container finished" podID="be427076-3fa7-46ee-89d1-c83bf42103f1" containerID="b18b97ae703cb75d188d16ad136702ed0938be75545ecccab7ba33be986544b4" exitCode=2
Sep 11 16:59:02.717658 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:02.717513    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm" event={"ID":"be427076-3fa7-46ee-89d1-c83bf42103f1","Type":"ContainerDied","Data":"b18b97ae703cb75d188d16ad136702ed0938be75545ecccab7ba33be986544b4"}
Sep 11 16:59:05.466495 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:05.466453    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm" podUID="be427076-3fa7-46ee-89d1-c83bf42103f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 11 16:59:05.473614 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:05.473583    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm" podUID="be427076-3fa7-46ee-89d1-c83bf42103f1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.46:8080/v2/models/isvc-xgboost-v2-runtime/ready\": dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 11 16:59:09.357860 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.357830    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:59:09.501333 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.501289    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbk6l\" (UniqueName: \"kubernetes.io/projected/be427076-3fa7-46ee-89d1-c83bf42103f1-kube-api-access-sbk6l\") pod \"be427076-3fa7-46ee-89d1-c83bf42103f1\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") "
Sep 11 16:59:09.501524 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.501351    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be427076-3fa7-46ee-89d1-c83bf42103f1-proxy-tls\") pod \"be427076-3fa7-46ee-89d1-c83bf42103f1\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") "
Sep 11 16:59:09.501524 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.501396    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be427076-3fa7-46ee-89d1-c83bf42103f1-kube-rbac-proxy-sar-config\") pod \"be427076-3fa7-46ee-89d1-c83bf42103f1\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") "
Sep 11 16:59:09.501524 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.501457    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be427076-3fa7-46ee-89d1-c83bf42103f1-cabundle-cert\") pod \"be427076-3fa7-46ee-89d1-c83bf42103f1\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") "
Sep 11 16:59:09.501524 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.501506    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be427076-3fa7-46ee-89d1-c83bf42103f1-kserve-provision-location\") pod \"be427076-3fa7-46ee-89d1-c83bf42103f1\" (UID: \"be427076-3fa7-46ee-89d1-c83bf42103f1\") "
Sep 11 16:59:09.501814 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.501783    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be427076-3fa7-46ee-89d1-c83bf42103f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "be427076-3fa7-46ee-89d1-c83bf42103f1" (UID: "be427076-3fa7-46ee-89d1-c83bf42103f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:59:09.501930 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.501827    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be427076-3fa7-46ee-89d1-c83bf42103f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be427076-3fa7-46ee-89d1-c83bf42103f1" (UID: "be427076-3fa7-46ee-89d1-c83bf42103f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:59:09.501930 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.501878    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be427076-3fa7-46ee-89d1-c83bf42103f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be427076-3fa7-46ee-89d1-c83bf42103f1" (UID: "be427076-3fa7-46ee-89d1-c83bf42103f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:59:09.503543 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.503522    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be427076-3fa7-46ee-89d1-c83bf42103f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be427076-3fa7-46ee-89d1-c83bf42103f1" (UID: "be427076-3fa7-46ee-89d1-c83bf42103f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:59:09.503617 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.503527    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be427076-3fa7-46ee-89d1-c83bf42103f1-kube-api-access-sbk6l" (OuterVolumeSpecName: "kube-api-access-sbk6l") pod "be427076-3fa7-46ee-89d1-c83bf42103f1" (UID: "be427076-3fa7-46ee-89d1-c83bf42103f1"). InnerVolumeSpecName "kube-api-access-sbk6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:59:09.602466 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.602431    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be427076-3fa7-46ee-89d1-c83bf42103f1-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:59:09.602466 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.602463    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbk6l\" (UniqueName: \"kubernetes.io/projected/be427076-3fa7-46ee-89d1-c83bf42103f1-kube-api-access-sbk6l\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:59:09.602661 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.602477    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be427076-3fa7-46ee-89d1-c83bf42103f1-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:59:09.602661 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.602492    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be427076-3fa7-46ee-89d1-c83bf42103f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:59:09.602661 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.602502    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be427076-3fa7-46ee-89d1-c83bf42103f1-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 16:59:09.752943 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.752852    2561 generic.go:356] "Generic (PLEG): container finished" podID="be427076-3fa7-46ee-89d1-c83bf42103f1" containerID="72c05d4b3f9fc816c0e811aebc682bcb643633481bb5ef09077982b0b17c3507" exitCode=0
Sep 11 16:59:09.752943 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.752908    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"
Sep 11 16:59:09.753114 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.752937    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm" event={"ID":"be427076-3fa7-46ee-89d1-c83bf42103f1","Type":"ContainerDied","Data":"72c05d4b3f9fc816c0e811aebc682bcb643633481bb5ef09077982b0b17c3507"}
Sep 11 16:59:09.753114 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.753006    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm" event={"ID":"be427076-3fa7-46ee-89d1-c83bf42103f1","Type":"ContainerDied","Data":"b56820a8a16b186a54514340e9f3011e10ee7437d82136118d939602b219b916"}
Sep 11 16:59:09.753114 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.753035    2561 scope.go:117] "RemoveContainer" containerID="b18b97ae703cb75d188d16ad136702ed0938be75545ecccab7ba33be986544b4"
Sep 11 16:59:09.762518 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.762496    2561 scope.go:117] "RemoveContainer" containerID="72c05d4b3f9fc816c0e811aebc682bcb643633481bb5ef09077982b0b17c3507"
Sep 11 16:59:09.770338 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.770320    2561 scope.go:117] "RemoveContainer" containerID="1eec88f1ef369b6a4f44ff40b6e5b8bfeafc343a03c17199c13469c6d6b56639"
Sep 11 16:59:09.779218 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.779198    2561 scope.go:117] "RemoveContainer" containerID="b18b97ae703cb75d188d16ad136702ed0938be75545ecccab7ba33be986544b4"
Sep 11 16:59:09.779518 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:59:09.779492    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18b97ae703cb75d188d16ad136702ed0938be75545ecccab7ba33be986544b4\": container with ID starting with b18b97ae703cb75d188d16ad136702ed0938be75545ecccab7ba33be986544b4 not found: ID does not exist" containerID="b18b97ae703cb75d188d16ad136702ed0938be75545ecccab7ba33be986544b4"
Sep 11 16:59:09.779617 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.779522    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18b97ae703cb75d188d16ad136702ed0938be75545ecccab7ba33be986544b4"} err="failed to get container status \"b18b97ae703cb75d188d16ad136702ed0938be75545ecccab7ba33be986544b4\": rpc error: code = NotFound desc = could not find container \"b18b97ae703cb75d188d16ad136702ed0938be75545ecccab7ba33be986544b4\": container with ID starting with b18b97ae703cb75d188d16ad136702ed0938be75545ecccab7ba33be986544b4 not found: ID does not exist"
Sep 11 16:59:09.779617 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.779541    2561 scope.go:117] "RemoveContainer" containerID="72c05d4b3f9fc816c0e811aebc682bcb643633481bb5ef09077982b0b17c3507"
Sep 11 16:59:09.779784 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:59:09.779764    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72c05d4b3f9fc816c0e811aebc682bcb643633481bb5ef09077982b0b17c3507\": container with ID starting with 72c05d4b3f9fc816c0e811aebc682bcb643633481bb5ef09077982b0b17c3507 not found: ID does not exist" containerID="72c05d4b3f9fc816c0e811aebc682bcb643633481bb5ef09077982b0b17c3507"
Sep 11 16:59:09.779859 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.779794    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c05d4b3f9fc816c0e811aebc682bcb643633481bb5ef09077982b0b17c3507"} err="failed to get container status \"72c05d4b3f9fc816c0e811aebc682bcb643633481bb5ef09077982b0b17c3507\": rpc error: code = NotFound desc = could not find container \"72c05d4b3f9fc816c0e811aebc682bcb643633481bb5ef09077982b0b17c3507\": container with ID starting with 72c05d4b3f9fc816c0e811aebc682bcb643633481bb5ef09077982b0b17c3507 not found: ID does not exist"
Sep 11 16:59:09.779859 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.779818    2561 scope.go:117] "RemoveContainer" containerID="1eec88f1ef369b6a4f44ff40b6e5b8bfeafc343a03c17199c13469c6d6b56639"
Sep 11 16:59:09.780042 ip-10-0-142-248 kubenswrapper[2561]: E0911 16:59:09.780025    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eec88f1ef369b6a4f44ff40b6e5b8bfeafc343a03c17199c13469c6d6b56639\": container with ID starting with 1eec88f1ef369b6a4f44ff40b6e5b8bfeafc343a03c17199c13469c6d6b56639 not found: ID does not exist" containerID="1eec88f1ef369b6a4f44ff40b6e5b8bfeafc343a03c17199c13469c6d6b56639"
Sep 11 16:59:09.780085 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.780047    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eec88f1ef369b6a4f44ff40b6e5b8bfeafc343a03c17199c13469c6d6b56639"} err="failed to get container status \"1eec88f1ef369b6a4f44ff40b6e5b8bfeafc343a03c17199c13469c6d6b56639\": rpc error: code = NotFound desc = could not find container \"1eec88f1ef369b6a4f44ff40b6e5b8bfeafc343a03c17199c13469c6d6b56639\": container with ID starting with 1eec88f1ef369b6a4f44ff40b6e5b8bfeafc343a03c17199c13469c6d6b56639 not found: ID does not exist"
Sep 11 16:59:09.794976 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.794945    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"]
Sep 11 16:59:09.805064 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:09.805037    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-f92tm"]
Sep 11 16:59:11.321871 ip-10-0-142-248 kubenswrapper[2561]: I0911 16:59:11.321838    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be427076-3fa7-46ee-89d1-c83bf42103f1" path="/var/lib/kubelet/pods/be427076-3fa7-46ee-89d1-c83bf42103f1/volumes"
Sep 11 17:00:56.115827 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.115789    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"]
Sep 11 17:00:56.116331 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.116142    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be427076-3fa7-46ee-89d1-c83bf42103f1" containerName="kube-rbac-proxy"
Sep 11 17:00:56.116331 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.116153    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="be427076-3fa7-46ee-89d1-c83bf42103f1" containerName="kube-rbac-proxy"
Sep 11 17:00:56.116331 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.116184    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be427076-3fa7-46ee-89d1-c83bf42103f1" containerName="storage-initializer"
Sep 11 17:00:56.116331 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.116190    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="be427076-3fa7-46ee-89d1-c83bf42103f1" containerName="storage-initializer"
Sep 11 17:00:56.116331 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.116198    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be427076-3fa7-46ee-89d1-c83bf42103f1" containerName="kserve-container"
Sep 11 17:00:56.116331 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.116203    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="be427076-3fa7-46ee-89d1-c83bf42103f1" containerName="kserve-container"
Sep 11 17:00:56.116331 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.116261    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="be427076-3fa7-46ee-89d1-c83bf42103f1" containerName="kube-rbac-proxy"
Sep 11 17:00:56.116331 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.116273    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="be427076-3fa7-46ee-89d1-c83bf42103f1" containerName="kserve-container"
Sep 11 17:00:56.119668 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.119648    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.122265 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.122242    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 11 17:00:56.122265 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.122258    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 11 17:00:56.122489 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.122305    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:00:56.122489 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.122366    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 17:00:56.122489 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.122422    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 17:00:56.122705 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.122602    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 17:00:56.122705 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.122672    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 17:00:56.131653 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.131628    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"]
Sep 11 17:00:56.218526 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.218488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcf488ca-0d6c-4981-8116-a10b4b012ca1-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.218707 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.218542    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lxwm\" (UniqueName: \"kubernetes.io/projected/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kube-api-access-7lxwm\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.218707 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.218621    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcf488ca-0d6c-4981-8116-a10b4b012ca1-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.218707 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.218654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.218707 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.218698    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.319779 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.319740    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lxwm\" (UniqueName: \"kubernetes.io/projected/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kube-api-access-7lxwm\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.319966 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.319800    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcf488ca-0d6c-4981-8116-a10b4b012ca1-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.319966 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.319836    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.319966 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.319879    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.319966 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.319902    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcf488ca-0d6c-4981-8116-a10b4b012ca1-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.320328 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.320304    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.320652 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.320631    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcf488ca-0d6c-4981-8116-a10b4b012ca1-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.320740 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.320713    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.322354 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.322333    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcf488ca-0d6c-4981-8116-a10b4b012ca1-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.328897 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.328875    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lxwm\" (UniqueName: \"kubernetes.io/projected/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kube-api-access-7lxwm\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.431796 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.431757    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:00:56.707883 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:56.707850    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"]
Sep 11 17:00:56.710410 ip-10-0-142-248 kubenswrapper[2561]: W0911 17:00:56.710378    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf488ca_0d6c_4981_8116_a10b4b012ca1.slice/crio-eea749a5f68386e3ded167a3a0f77fe3da55bdc274ea89f5e931859f26faba12 WatchSource:0}: Error finding container eea749a5f68386e3ded167a3a0f77fe3da55bdc274ea89f5e931859f26faba12: Status 404 returned error can't find the container with id eea749a5f68386e3ded167a3a0f77fe3da55bdc274ea89f5e931859f26faba12
Sep 11 17:00:57.223858 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:57.223806    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f" event={"ID":"bcf488ca-0d6c-4981-8116-a10b4b012ca1","Type":"ContainerStarted","Data":"fff5909faf10e868ceb07b378f96d5386252b9bed1283e2a8b88fa248ec88d1f"}
Sep 11 17:00:57.224301 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:00:57.223895    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f" event={"ID":"bcf488ca-0d6c-4981-8116-a10b4b012ca1","Type":"ContainerStarted","Data":"eea749a5f68386e3ded167a3a0f77fe3da55bdc274ea89f5e931859f26faba12"}
Sep 11 17:01:01.244863 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:01.244826    2561 generic.go:356] "Generic (PLEG): container finished" podID="bcf488ca-0d6c-4981-8116-a10b4b012ca1" containerID="fff5909faf10e868ceb07b378f96d5386252b9bed1283e2a8b88fa248ec88d1f" exitCode=0
Sep 11 17:01:01.245329 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:01.244878    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f" event={"ID":"bcf488ca-0d6c-4981-8116-a10b4b012ca1","Type":"ContainerDied","Data":"fff5909faf10e868ceb07b378f96d5386252b9bed1283e2a8b88fa248ec88d1f"}
Sep 11 17:01:02.253085 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:02.253042    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f" event={"ID":"bcf488ca-0d6c-4981-8116-a10b4b012ca1","Type":"ContainerStarted","Data":"4dde991d380b7443d2f711c6c51511d11556855cef0d13695d2f1935ebfe691d"}
Sep 11 17:01:02.253487 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:02.253127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f" event={"ID":"bcf488ca-0d6c-4981-8116-a10b4b012ca1","Type":"ContainerStarted","Data":"23b724356da6a046f0ea14f0903934b4e713229240c2ed10faacd5afe8f91556"}
Sep 11 17:01:02.253487 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:02.253382    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:01:02.282946 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:02.282894    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f" podStartSLOduration=2.457953542 podStartE2EDuration="6.282876823s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.824923281s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:01:02.28116764 +0000 UTC m=+2667.621261723" watchObservedRunningTime="2026-09-11 17:01:02.282876823 +0000 UTC m=+2667.622970868"
Sep 11 17:01:03.273153 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:03.273113    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:01:09.283838 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:09.283808    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:01:35.378343 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:35.378312    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 17:01:35.491821 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:35.491797    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 17:01:39.287622 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:39.287591    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:01:46.169868 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:46.169823    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"]
Sep 11 17:01:46.170312 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:46.170075    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f" podUID="bcf488ca-0d6c-4981-8116-a10b4b012ca1" containerName="kserve-container" containerID="cri-o://23b724356da6a046f0ea14f0903934b4e713229240c2ed10faacd5afe8f91556" gracePeriod=30
Sep 11 17:01:46.170312 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:46.170134    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f" podUID="bcf488ca-0d6c-4981-8116-a10b4b012ca1" containerName="kube-rbac-proxy" containerID="cri-o://4dde991d380b7443d2f711c6c51511d11556855cef0d13695d2f1935ebfe691d" gracePeriod=30
Sep 11 17:01:46.481036 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:46.480935    2561 generic.go:356] "Generic (PLEG): container finished" podID="bcf488ca-0d6c-4981-8116-a10b4b012ca1" containerID="4dde991d380b7443d2f711c6c51511d11556855cef0d13695d2f1935ebfe691d" exitCode=2
Sep 11 17:01:46.481036 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:46.480989    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f" event={"ID":"bcf488ca-0d6c-4981-8116-a10b4b012ca1","Type":"ContainerDied","Data":"4dde991d380b7443d2f711c6c51511d11556855cef0d13695d2f1935ebfe691d"}
Sep 11 17:01:47.322146 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.322121    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:01:47.455779 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.455748    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcf488ca-0d6c-4981-8116-a10b4b012ca1-proxy-tls\") pod \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") "
Sep 11 17:01:47.455966 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.455812    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lxwm\" (UniqueName: \"kubernetes.io/projected/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kube-api-access-7lxwm\") pod \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") "
Sep 11 17:01:47.455966 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.455842    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcf488ca-0d6c-4981-8116-a10b4b012ca1-cabundle-cert\") pod \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") "
Sep 11 17:01:47.455966 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.455868    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kserve-provision-location\") pod \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") "
Sep 11 17:01:47.455966 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.455918    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kube-rbac-proxy-sar-config\") pod \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\" (UID: \"bcf488ca-0d6c-4981-8116-a10b4b012ca1\") "
Sep 11 17:01:47.456788 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.456303    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bcf488ca-0d6c-4981-8116-a10b4b012ca1" (UID: "bcf488ca-0d6c-4981-8116-a10b4b012ca1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:01:47.456788 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.456303    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf488ca-0d6c-4981-8116-a10b4b012ca1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bcf488ca-0d6c-4981-8116-a10b4b012ca1" (UID: "bcf488ca-0d6c-4981-8116-a10b4b012ca1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:01:47.456788 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.456340    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bcf488ca-0d6c-4981-8116-a10b4b012ca1" (UID: "bcf488ca-0d6c-4981-8116-a10b4b012ca1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:01:47.458170 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.458146    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kube-api-access-7lxwm" (OuterVolumeSpecName: "kube-api-access-7lxwm") pod "bcf488ca-0d6c-4981-8116-a10b4b012ca1" (UID: "bcf488ca-0d6c-4981-8116-a10b4b012ca1"). InnerVolumeSpecName "kube-api-access-7lxwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:01:47.458260 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.458196    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf488ca-0d6c-4981-8116-a10b4b012ca1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bcf488ca-0d6c-4981-8116-a10b4b012ca1" (UID: "bcf488ca-0d6c-4981-8116-a10b4b012ca1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:01:47.487280 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.487250    2561 generic.go:356] "Generic (PLEG): container finished" podID="bcf488ca-0d6c-4981-8116-a10b4b012ca1" containerID="23b724356da6a046f0ea14f0903934b4e713229240c2ed10faacd5afe8f91556" exitCode=0
Sep 11 17:01:47.487438 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.487309    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"
Sep 11 17:01:47.487438 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.487314    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f" event={"ID":"bcf488ca-0d6c-4981-8116-a10b4b012ca1","Type":"ContainerDied","Data":"23b724356da6a046f0ea14f0903934b4e713229240c2ed10faacd5afe8f91556"}
Sep 11 17:01:47.487512 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.487456    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f" event={"ID":"bcf488ca-0d6c-4981-8116-a10b4b012ca1","Type":"ContainerDied","Data":"eea749a5f68386e3ded167a3a0f77fe3da55bdc274ea89f5e931859f26faba12"}
Sep 11 17:01:47.487512 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.487491    2561 scope.go:117] "RemoveContainer" containerID="4dde991d380b7443d2f711c6c51511d11556855cef0d13695d2f1935ebfe691d"
Sep 11 17:01:47.496539 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.496518    2561 scope.go:117] "RemoveContainer" containerID="23b724356da6a046f0ea14f0903934b4e713229240c2ed10faacd5afe8f91556"
Sep 11 17:01:47.504207 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.504186    2561 scope.go:117] "RemoveContainer" containerID="fff5909faf10e868ceb07b378f96d5386252b9bed1283e2a8b88fa248ec88d1f"
Sep 11 17:01:47.511797 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.511766    2561 scope.go:117] "RemoveContainer" containerID="4dde991d380b7443d2f711c6c51511d11556855cef0d13695d2f1935ebfe691d"
Sep 11 17:01:47.512159 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:01:47.512135    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dde991d380b7443d2f711c6c51511d11556855cef0d13695d2f1935ebfe691d\": container with ID starting with 4dde991d380b7443d2f711c6c51511d11556855cef0d13695d2f1935ebfe691d not found: ID does not exist" containerID="4dde991d380b7443d2f711c6c51511d11556855cef0d13695d2f1935ebfe691d"
Sep 11 17:01:47.512268 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.512166    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dde991d380b7443d2f711c6c51511d11556855cef0d13695d2f1935ebfe691d"} err="failed to get container status \"4dde991d380b7443d2f711c6c51511d11556855cef0d13695d2f1935ebfe691d\": rpc error: code = NotFound desc = could not find container \"4dde991d380b7443d2f711c6c51511d11556855cef0d13695d2f1935ebfe691d\": container with ID starting with 4dde991d380b7443d2f711c6c51511d11556855cef0d13695d2f1935ebfe691d not found: ID does not exist"
Sep 11 17:01:47.512268 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.512210    2561 scope.go:117] "RemoveContainer" containerID="23b724356da6a046f0ea14f0903934b4e713229240c2ed10faacd5afe8f91556"
Sep 11 17:01:47.512494 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:01:47.512478    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b724356da6a046f0ea14f0903934b4e713229240c2ed10faacd5afe8f91556\": container with ID starting with 23b724356da6a046f0ea14f0903934b4e713229240c2ed10faacd5afe8f91556 not found: ID does not exist" containerID="23b724356da6a046f0ea14f0903934b4e713229240c2ed10faacd5afe8f91556"
Sep 11 17:01:47.512548 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.512500    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b724356da6a046f0ea14f0903934b4e713229240c2ed10faacd5afe8f91556"} err="failed to get container status \"23b724356da6a046f0ea14f0903934b4e713229240c2ed10faacd5afe8f91556\": rpc error: code = NotFound desc = could not find container \"23b724356da6a046f0ea14f0903934b4e713229240c2ed10faacd5afe8f91556\": container with ID starting with 23b724356da6a046f0ea14f0903934b4e713229240c2ed10faacd5afe8f91556 not found: ID does not exist"
Sep 11 17:01:47.512548 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.512514    2561 scope.go:117] "RemoveContainer" containerID="fff5909faf10e868ceb07b378f96d5386252b9bed1283e2a8b88fa248ec88d1f"
Sep 11 17:01:47.512786 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:01:47.512767    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff5909faf10e868ceb07b378f96d5386252b9bed1283e2a8b88fa248ec88d1f\": container with ID starting with fff5909faf10e868ceb07b378f96d5386252b9bed1283e2a8b88fa248ec88d1f not found: ID does not exist" containerID="fff5909faf10e868ceb07b378f96d5386252b9bed1283e2a8b88fa248ec88d1f"
Sep 11 17:01:47.512847 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.512797    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff5909faf10e868ceb07b378f96d5386252b9bed1283e2a8b88fa248ec88d1f"} err="failed to get container status \"fff5909faf10e868ceb07b378f96d5386252b9bed1283e2a8b88fa248ec88d1f\": rpc error: code = NotFound desc = could not find container \"fff5909faf10e868ceb07b378f96d5386252b9bed1283e2a8b88fa248ec88d1f\": container with ID starting with fff5909faf10e868ceb07b378f96d5386252b9bed1283e2a8b88fa248ec88d1f not found: ID does not exist"
Sep 11 17:01:47.513630 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.513608    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"]
Sep 11 17:01:47.517817 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.517796    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-ltw2f"]
Sep 11 17:01:47.557556 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.557522    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:01:47.557556 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.557555    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:01:47.557725 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.557570    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcf488ca-0d6c-4981-8116-a10b4b012ca1-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:01:47.557725 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.557583    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lxwm\" (UniqueName: \"kubernetes.io/projected/bcf488ca-0d6c-4981-8116-a10b4b012ca1-kube-api-access-7lxwm\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:01:47.557725 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:47.557595    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcf488ca-0d6c-4981-8116-a10b4b012ca1-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:01:49.321015 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:01:49.320979    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcf488ca-0d6c-4981-8116-a10b4b012ca1" path="/var/lib/kubelet/pods/bcf488ca-0d6c-4981-8116-a10b4b012ca1/volumes"
Sep 11 17:02:02.606318 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.606282    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"]
Sep 11 17:02:02.606720 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.606657    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf488ca-0d6c-4981-8116-a10b4b012ca1" containerName="storage-initializer"
Sep 11 17:02:02.606720 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.606672    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf488ca-0d6c-4981-8116-a10b4b012ca1" containerName="storage-initializer"
Sep 11 17:02:02.606720 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.606686    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf488ca-0d6c-4981-8116-a10b4b012ca1" containerName="kube-rbac-proxy"
Sep 11 17:02:02.606720 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.606692    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf488ca-0d6c-4981-8116-a10b4b012ca1" containerName="kube-rbac-proxy"
Sep 11 17:02:02.606720 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.606700    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf488ca-0d6c-4981-8116-a10b4b012ca1" containerName="kserve-container"
Sep 11 17:02:02.606720 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.606706    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf488ca-0d6c-4981-8116-a10b4b012ca1" containerName="kserve-container"
Sep 11 17:02:02.607006 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.606831    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcf488ca-0d6c-4981-8116-a10b4b012ca1" containerName="kserve-container"
Sep 11 17:02:02.607006 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.606849    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcf488ca-0d6c-4981-8116-a10b4b012ca1" containerName="kube-rbac-proxy"
Sep 11 17:02:02.611783 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.611762    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:02.614168 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.613964    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:02:02.614168 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.613973    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 17:02:02.614168 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.614001    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 17:02:02.614168 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.614041    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 11 17:02:02.614168 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.613973    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 11 17:02:02.614168 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.614161    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 11 17:02:02.614502 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.614269    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 17:02:02.621099 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.621079    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"]
Sep 11 17:02:02.799883 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.799848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73b13958-47c7-4dde-a8ff-d6728f1b7146-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:02.800081 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.799912    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73b13958-47c7-4dde-a8ff-d6728f1b7146-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:02.800081 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.799940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73b13958-47c7-4dde-a8ff-d6728f1b7146-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:02.800081 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.799964    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjzfk\" (UniqueName: \"kubernetes.io/projected/73b13958-47c7-4dde-a8ff-d6728f1b7146-kube-api-access-cjzfk\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:02.800081 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.799985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73b13958-47c7-4dde-a8ff-d6728f1b7146-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:02.900793 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.900755    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73b13958-47c7-4dde-a8ff-d6728f1b7146-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:02.900957 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.900814    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjzfk\" (UniqueName: \"kubernetes.io/projected/73b13958-47c7-4dde-a8ff-d6728f1b7146-kube-api-access-cjzfk\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:02.900957 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.900846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73b13958-47c7-4dde-a8ff-d6728f1b7146-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:02.900957 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.900885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73b13958-47c7-4dde-a8ff-d6728f1b7146-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:02.901137 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.900969    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73b13958-47c7-4dde-a8ff-d6728f1b7146-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:02.901324 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.901298    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73b13958-47c7-4dde-a8ff-d6728f1b7146-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:02.901556 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.901536    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73b13958-47c7-4dde-a8ff-d6728f1b7146-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:02.901633 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.901594    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73b13958-47c7-4dde-a8ff-d6728f1b7146-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:02.903573 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.903555    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73b13958-47c7-4dde-a8ff-d6728f1b7146-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:02.910808 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.910786    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjzfk\" (UniqueName: \"kubernetes.io/projected/73b13958-47c7-4dde-a8ff-d6728f1b7146-kube-api-access-cjzfk\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:02.922665 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:02.922633    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:03.197973 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:03.197947    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"]
Sep 11 17:02:03.200493 ip-10-0-142-248 kubenswrapper[2561]: W0911 17:02:03.200464    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b13958_47c7_4dde_a8ff_d6728f1b7146.slice/crio-4e3a1da87cfc00143d5ae59e022f47c755d0e20bdf85f664f3c40db048689c51 WatchSource:0}: Error finding container 4e3a1da87cfc00143d5ae59e022f47c755d0e20bdf85f664f3c40db048689c51: Status 404 returned error can't find the container with id 4e3a1da87cfc00143d5ae59e022f47c755d0e20bdf85f664f3c40db048689c51
Sep 11 17:02:03.564215 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:03.564092    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" event={"ID":"73b13958-47c7-4dde-a8ff-d6728f1b7146","Type":"ContainerStarted","Data":"ae03ee2f2bb5b3f2b2e0fb7b4f35755a8772ce14305d148f69cef1656c982cd3"}
Sep 11 17:02:03.564215 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:03.564169    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" event={"ID":"73b13958-47c7-4dde-a8ff-d6728f1b7146","Type":"ContainerStarted","Data":"4e3a1da87cfc00143d5ae59e022f47c755d0e20bdf85f664f3c40db048689c51"}
Sep 11 17:02:04.576719 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:04.576676    2561 generic.go:356] "Generic (PLEG): container finished" podID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerID="ae03ee2f2bb5b3f2b2e0fb7b4f35755a8772ce14305d148f69cef1656c982cd3" exitCode=0
Sep 11 17:02:04.577196 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:04.576733    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" event={"ID":"73b13958-47c7-4dde-a8ff-d6728f1b7146","Type":"ContainerDied","Data":"ae03ee2f2bb5b3f2b2e0fb7b4f35755a8772ce14305d148f69cef1656c982cd3"}
Sep 11 17:02:05.584114 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:05.584071    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" event={"ID":"73b13958-47c7-4dde-a8ff-d6728f1b7146","Type":"ContainerStarted","Data":"321a3ea9e225b1da2a45772e93a49486132fbc61894a375fc03d416794347fd5"}
Sep 11 17:02:05.584536 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:05.584148    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" event={"ID":"73b13958-47c7-4dde-a8ff-d6728f1b7146","Type":"ContainerStarted","Data":"186e7849545da7987a5f2bd6601b724052c155722b49027083d929516bea5ea5"}
Sep 11 17:02:05.584536 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:05.584355    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:05.584536 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:05.584488    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:05.585751 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:05.585723    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 11 17:02:05.609383 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:05.609336    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" podStartSLOduration=2.400478484 podStartE2EDuration="3.609320897s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.208842413s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:02:05.60761421 +0000 UTC m=+2730.947708255" watchObservedRunningTime="2026-09-11 17:02:05.609320897 +0000 UTC m=+2730.949414944"
Sep 11 17:02:06.588575 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:06.588534    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 11 17:02:11.593490 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:11.593460    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:02:11.593943 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:11.593917    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 11 17:02:21.594407 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:21.594367    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 11 17:02:26.635308 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.635272    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"]
Sep 11 17:02:26.641388 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.641352    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:26.644518 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.644488    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 11 17:02:26.644683 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.644488    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 11 17:02:26.644683 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.644492    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 17:02:26.649910 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.649881    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"]
Sep 11 17:02:26.713442 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.713406    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:26.713442 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.713442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:26.713686 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.713472    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:26.713686 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.713562    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:26.713686 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.713635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5s2m\" (UniqueName: \"kubernetes.io/projected/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kube-api-access-r5s2m\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:26.815023 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.814979    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:26.815258 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.815040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:26.815258 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.815114    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5s2m\" (UniqueName: \"kubernetes.io/projected/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kube-api-access-r5s2m\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:26.815258 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.815163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:26.815258 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.815219    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:26.815600 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.815578    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:26.815793 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.815773    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:26.815865 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.815846    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:26.817631 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.817612    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:26.824495 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.824468    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5s2m\" (UniqueName: \"kubernetes.io/projected/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kube-api-access-r5s2m\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:26.955509 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:26.955466    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:27.229854 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:27.229828    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"]
Sep 11 17:02:27.231885 ip-10-0-142-248 kubenswrapper[2561]: W0911 17:02:27.231854    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8493cf83_2eeb_486e_a6fe_5384b8d7ffae.slice/crio-eac41a00542afdcd1a1853c5df811a4962d3fc0c5e87ce62e94f53cd57d3da15 WatchSource:0}: Error finding container eac41a00542afdcd1a1853c5df811a4962d3fc0c5e87ce62e94f53cd57d3da15: Status 404 returned error can't find the container with id eac41a00542afdcd1a1853c5df811a4962d3fc0c5e87ce62e94f53cd57d3da15
Sep 11 17:02:27.691262 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:27.691220    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8" event={"ID":"8493cf83-2eeb-486e-a6fe-5384b8d7ffae","Type":"ContainerStarted","Data":"4ea93031b4f4bf9fb1edc288bcfe7cc9dd53600aa6d9b3db8231861c07ff60d1"}
Sep 11 17:02:27.691666 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:27.691285    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8" event={"ID":"8493cf83-2eeb-486e-a6fe-5384b8d7ffae","Type":"ContainerStarted","Data":"eac41a00542afdcd1a1853c5df811a4962d3fc0c5e87ce62e94f53cd57d3da15"}
Sep 11 17:02:31.594606 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:31.594562    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 11 17:02:31.712365 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:31.712331    2561 generic.go:356] "Generic (PLEG): container finished" podID="8493cf83-2eeb-486e-a6fe-5384b8d7ffae" containerID="4ea93031b4f4bf9fb1edc288bcfe7cc9dd53600aa6d9b3db8231861c07ff60d1" exitCode=0
Sep 11 17:02:31.712569 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:31.712405    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8" event={"ID":"8493cf83-2eeb-486e-a6fe-5384b8d7ffae","Type":"ContainerDied","Data":"4ea93031b4f4bf9fb1edc288bcfe7cc9dd53600aa6d9b3db8231861c07ff60d1"}
Sep 11 17:02:32.719165 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:32.719128    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8" event={"ID":"8493cf83-2eeb-486e-a6fe-5384b8d7ffae","Type":"ContainerStarted","Data":"91a2298a5fb7bf70f6f1ed68b111a9fe0609e87dc6baf583c10cfb5216c235ea"}
Sep 11 17:02:32.719619 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:32.719234    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8" event={"ID":"8493cf83-2eeb-486e-a6fe-5384b8d7ffae","Type":"ContainerStarted","Data":"3e88587f8aba8fafa538c37fd83006a0eeefa835ccccccf493b803e069cd0e46"}
Sep 11 17:02:32.719619 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:32.719441    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:32.719619 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:32.719488    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:32.749967 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:32.749913    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8" podStartSLOduration=2.673068055 podStartE2EDuration="6.749897177s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.076829122s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:02:32.747914112 +0000 UTC m=+2758.088008171" watchObservedRunningTime="2026-09-11 17:02:32.749897177 +0000 UTC m=+2758.089991224"
Sep 11 17:02:38.729563 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:38.729531    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:02:41.594066 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:41.594026    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 11 17:02:51.594779 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:02:51.594736    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 11 17:03:01.594026 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:01.593943    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 11 17:03:08.734123 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:08.734089    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:03:11.594190 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:11.594138    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 11 17:03:16.690394 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:16.690362    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"]
Sep 11 17:03:16.692783 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:16.690576    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8" podUID="8493cf83-2eeb-486e-a6fe-5384b8d7ffae" containerName="kserve-container" containerID="cri-o://3e88587f8aba8fafa538c37fd83006a0eeefa835ccccccf493b803e069cd0e46" gracePeriod=30
Sep 11 17:03:16.692783 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:16.690609    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8" podUID="8493cf83-2eeb-486e-a6fe-5384b8d7ffae" containerName="kube-rbac-proxy" containerID="cri-o://91a2298a5fb7bf70f6f1ed68b111a9fe0609e87dc6baf583c10cfb5216c235ea" gracePeriod=30
Sep 11 17:03:16.940231 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:16.940165    2561 generic.go:356] "Generic (PLEG): container finished" podID="8493cf83-2eeb-486e-a6fe-5384b8d7ffae" containerID="91a2298a5fb7bf70f6f1ed68b111a9fe0609e87dc6baf583c10cfb5216c235ea" exitCode=2
Sep 11 17:03:16.940443 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:16.940242    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8" event={"ID":"8493cf83-2eeb-486e-a6fe-5384b8d7ffae","Type":"ContainerDied","Data":"91a2298a5fb7bf70f6f1ed68b111a9fe0609e87dc6baf583c10cfb5216c235ea"}
Sep 11 17:03:17.948707 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:17.948666    2561 generic.go:356] "Generic (PLEG): container finished" podID="8493cf83-2eeb-486e-a6fe-5384b8d7ffae" containerID="3e88587f8aba8fafa538c37fd83006a0eeefa835ccccccf493b803e069cd0e46" exitCode=0
Sep 11 17:03:17.948707 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:17.948689    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8" event={"ID":"8493cf83-2eeb-486e-a6fe-5384b8d7ffae","Type":"ContainerDied","Data":"3e88587f8aba8fafa538c37fd83006a0eeefa835ccccccf493b803e069cd0e46"}
Sep 11 17:03:17.964552 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:17.964528    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:03:18.054021 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.053927    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kserve-provision-location\") pod \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") "
Sep 11 17:03:18.054021 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.053989    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kube-rbac-proxy-sar-config\") pod \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") "
Sep 11 17:03:18.054265 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.054095    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-proxy-tls\") pod \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") "
Sep 11 17:03:18.054265 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.054122    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-cabundle-cert\") pod \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") "
Sep 11 17:03:18.054265 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.054166    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5s2m\" (UniqueName: \"kubernetes.io/projected/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kube-api-access-r5s2m\") pod \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\" (UID: \"8493cf83-2eeb-486e-a6fe-5384b8d7ffae\") "
Sep 11 17:03:18.054429 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.054366    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8493cf83-2eeb-486e-a6fe-5384b8d7ffae" (UID: "8493cf83-2eeb-486e-a6fe-5384b8d7ffae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:03:18.054502 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.054375    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8493cf83-2eeb-486e-a6fe-5384b8d7ffae" (UID: "8493cf83-2eeb-486e-a6fe-5384b8d7ffae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:03:18.054561 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.054513    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:03:18.054615 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.054570    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8493cf83-2eeb-486e-a6fe-5384b8d7ffae" (UID: "8493cf83-2eeb-486e-a6fe-5384b8d7ffae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:03:18.056340 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.056308    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kube-api-access-r5s2m" (OuterVolumeSpecName: "kube-api-access-r5s2m") pod "8493cf83-2eeb-486e-a6fe-5384b8d7ffae" (UID: "8493cf83-2eeb-486e-a6fe-5384b8d7ffae"). InnerVolumeSpecName "kube-api-access-r5s2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:03:18.056462 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.056358    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8493cf83-2eeb-486e-a6fe-5384b8d7ffae" (UID: "8493cf83-2eeb-486e-a6fe-5384b8d7ffae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:03:18.155259 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.155222    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:03:18.155259 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.155251    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:03:18.155259 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.155259    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5s2m\" (UniqueName: \"kubernetes.io/projected/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kube-api-access-r5s2m\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:03:18.155259 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.155268    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8493cf83-2eeb-486e-a6fe-5384b8d7ffae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:03:18.955415 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.955392    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"
Sep 11 17:03:18.955834 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.955413    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8" event={"ID":"8493cf83-2eeb-486e-a6fe-5384b8d7ffae","Type":"ContainerDied","Data":"eac41a00542afdcd1a1853c5df811a4962d3fc0c5e87ce62e94f53cd57d3da15"}
Sep 11 17:03:18.955834 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.955479    2561 scope.go:117] "RemoveContainer" containerID="91a2298a5fb7bf70f6f1ed68b111a9fe0609e87dc6baf583c10cfb5216c235ea"
Sep 11 17:03:18.964129 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.964111    2561 scope.go:117] "RemoveContainer" containerID="3e88587f8aba8fafa538c37fd83006a0eeefa835ccccccf493b803e069cd0e46"
Sep 11 17:03:18.972891 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.972871    2561 scope.go:117] "RemoveContainer" containerID="4ea93031b4f4bf9fb1edc288bcfe7cc9dd53600aa6d9b3db8231861c07ff60d1"
Sep 11 17:03:18.981804 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.981782    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"]
Sep 11 17:03:18.987785 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:18.987761    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-wcft8"]
Sep 11 17:03:19.321578 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:19.321497    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8493cf83-2eeb-486e-a6fe-5384b8d7ffae" path="/var/lib/kubelet/pods/8493cf83-2eeb-486e-a6fe-5384b8d7ffae/volumes"
Sep 11 17:03:21.594979 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:21.594952    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:03:22.633510 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:22.633471    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"]
Sep 11 17:03:22.633977 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:22.633744    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kserve-container" containerID="cri-o://186e7849545da7987a5f2bd6601b724052c155722b49027083d929516bea5ea5" gracePeriod=30
Sep 11 17:03:22.633977 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:22.633802    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kube-rbac-proxy" containerID="cri-o://321a3ea9e225b1da2a45772e93a49486132fbc61894a375fc03d416794347fd5" gracePeriod=30
Sep 11 17:03:22.979334 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:22.979297    2561 generic.go:356] "Generic (PLEG): container finished" podID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerID="321a3ea9e225b1da2a45772e93a49486132fbc61894a375fc03d416794347fd5" exitCode=2
Sep 11 17:03:22.979504 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:22.979370    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" event={"ID":"73b13958-47c7-4dde-a8ff-d6728f1b7146","Type":"ContainerDied","Data":"321a3ea9e225b1da2a45772e93a49486132fbc61894a375fc03d416794347fd5"}
Sep 11 17:03:26.589771 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:26.589717    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 11 17:03:27.491533 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:27.491507    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:03:27.639377 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:27.639274    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjzfk\" (UniqueName: \"kubernetes.io/projected/73b13958-47c7-4dde-a8ff-d6728f1b7146-kube-api-access-cjzfk\") pod \"73b13958-47c7-4dde-a8ff-d6728f1b7146\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") "
Sep 11 17:03:27.639377 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:27.639320    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73b13958-47c7-4dde-a8ff-d6728f1b7146-kube-rbac-proxy-sar-config\") pod \"73b13958-47c7-4dde-a8ff-d6728f1b7146\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") "
Sep 11 17:03:27.639377 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:27.639369    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73b13958-47c7-4dde-a8ff-d6728f1b7146-proxy-tls\") pod \"73b13958-47c7-4dde-a8ff-d6728f1b7146\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") "
Sep 11 17:03:27.639915 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:27.639432    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73b13958-47c7-4dde-a8ff-d6728f1b7146-cabundle-cert\") pod \"73b13958-47c7-4dde-a8ff-d6728f1b7146\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") "
Sep 11 17:03:27.639915 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:27.639469    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73b13958-47c7-4dde-a8ff-d6728f1b7146-kserve-provision-location\") pod \"73b13958-47c7-4dde-a8ff-d6728f1b7146\" (UID: \"73b13958-47c7-4dde-a8ff-d6728f1b7146\") "
Sep 11 17:03:27.639915 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:27.639829    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b13958-47c7-4dde-a8ff-d6728f1b7146-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "73b13958-47c7-4dde-a8ff-d6728f1b7146" (UID: "73b13958-47c7-4dde-a8ff-d6728f1b7146"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:03:27.639915 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:27.639838    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b13958-47c7-4dde-a8ff-d6728f1b7146-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "73b13958-47c7-4dde-a8ff-d6728f1b7146" (UID: "73b13958-47c7-4dde-a8ff-d6728f1b7146"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:03:27.640094 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:27.639984    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b13958-47c7-4dde-a8ff-d6728f1b7146-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "73b13958-47c7-4dde-a8ff-d6728f1b7146" (UID: "73b13958-47c7-4dde-a8ff-d6728f1b7146"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:03:27.641470 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:27.641452    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73b13958-47c7-4dde-a8ff-d6728f1b7146-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73b13958-47c7-4dde-a8ff-d6728f1b7146" (UID: "73b13958-47c7-4dde-a8ff-d6728f1b7146"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:03:27.641563 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:27.641546    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b13958-47c7-4dde-a8ff-d6728f1b7146-kube-api-access-cjzfk" (OuterVolumeSpecName: "kube-api-access-cjzfk") pod "73b13958-47c7-4dde-a8ff-d6728f1b7146" (UID: "73b13958-47c7-4dde-a8ff-d6728f1b7146"). InnerVolumeSpecName "kube-api-access-cjzfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:03:27.740835 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:27.740795    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjzfk\" (UniqueName: \"kubernetes.io/projected/73b13958-47c7-4dde-a8ff-d6728f1b7146-kube-api-access-cjzfk\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:03:27.740835 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:27.740828    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73b13958-47c7-4dde-a8ff-d6728f1b7146-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:03:27.740835 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:27.740838    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73b13958-47c7-4dde-a8ff-d6728f1b7146-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:03:27.741074 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:27.740851    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73b13958-47c7-4dde-a8ff-d6728f1b7146-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:03:27.741074 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:27.740860    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73b13958-47c7-4dde-a8ff-d6728f1b7146-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:03:28.004118 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:28.004079    2561 generic.go:356] "Generic (PLEG): container finished" podID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerID="186e7849545da7987a5f2bd6601b724052c155722b49027083d929516bea5ea5" exitCode=0
Sep 11 17:03:28.004312 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:28.004144    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"
Sep 11 17:03:28.004312 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:28.004158    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" event={"ID":"73b13958-47c7-4dde-a8ff-d6728f1b7146","Type":"ContainerDied","Data":"186e7849545da7987a5f2bd6601b724052c155722b49027083d929516bea5ea5"}
Sep 11 17:03:28.004312 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:28.004238    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp" event={"ID":"73b13958-47c7-4dde-a8ff-d6728f1b7146","Type":"ContainerDied","Data":"4e3a1da87cfc00143d5ae59e022f47c755d0e20bdf85f664f3c40db048689c51"}
Sep 11 17:03:28.004312 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:28.004265    2561 scope.go:117] "RemoveContainer" containerID="321a3ea9e225b1da2a45772e93a49486132fbc61894a375fc03d416794347fd5"
Sep 11 17:03:28.013110 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:28.013091    2561 scope.go:117] "RemoveContainer" containerID="186e7849545da7987a5f2bd6601b724052c155722b49027083d929516bea5ea5"
Sep 11 17:03:28.020776 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:28.020750    2561 scope.go:117] "RemoveContainer" containerID="ae03ee2f2bb5b3f2b2e0fb7b4f35755a8772ce14305d148f69cef1656c982cd3"
Sep 11 17:03:28.028617 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:28.028594    2561 scope.go:117] "RemoveContainer" containerID="321a3ea9e225b1da2a45772e93a49486132fbc61894a375fc03d416794347fd5"
Sep 11 17:03:28.028936 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:03:28.028913    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321a3ea9e225b1da2a45772e93a49486132fbc61894a375fc03d416794347fd5\": container with ID starting with 321a3ea9e225b1da2a45772e93a49486132fbc61894a375fc03d416794347fd5 not found: ID does not exist" containerID="321a3ea9e225b1da2a45772e93a49486132fbc61894a375fc03d416794347fd5"
Sep 11 17:03:28.028995 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:28.028950    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321a3ea9e225b1da2a45772e93a49486132fbc61894a375fc03d416794347fd5"} err="failed to get container status \"321a3ea9e225b1da2a45772e93a49486132fbc61894a375fc03d416794347fd5\": rpc error: code = NotFound desc = could not find container \"321a3ea9e225b1da2a45772e93a49486132fbc61894a375fc03d416794347fd5\": container with ID starting with 321a3ea9e225b1da2a45772e93a49486132fbc61894a375fc03d416794347fd5 not found: ID does not exist"
Sep 11 17:03:28.028995 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:28.028975    2561 scope.go:117] "RemoveContainer" containerID="186e7849545da7987a5f2bd6601b724052c155722b49027083d929516bea5ea5"
Sep 11 17:03:28.029311 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:03:28.029292    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186e7849545da7987a5f2bd6601b724052c155722b49027083d929516bea5ea5\": container with ID starting with 186e7849545da7987a5f2bd6601b724052c155722b49027083d929516bea5ea5 not found: ID does not exist" containerID="186e7849545da7987a5f2bd6601b724052c155722b49027083d929516bea5ea5"
Sep 11 17:03:28.029370 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:28.029317    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186e7849545da7987a5f2bd6601b724052c155722b49027083d929516bea5ea5"} err="failed to get container status \"186e7849545da7987a5f2bd6601b724052c155722b49027083d929516bea5ea5\": rpc error: code = NotFound desc = could not find container \"186e7849545da7987a5f2bd6601b724052c155722b49027083d929516bea5ea5\": container with ID starting with 186e7849545da7987a5f2bd6601b724052c155722b49027083d929516bea5ea5 not found: ID does not exist"
Sep 11 17:03:28.029370 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:28.029333    2561 scope.go:117] "RemoveContainer" containerID="ae03ee2f2bb5b3f2b2e0fb7b4f35755a8772ce14305d148f69cef1656c982cd3"
Sep 11 17:03:28.029659 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:03:28.029636    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae03ee2f2bb5b3f2b2e0fb7b4f35755a8772ce14305d148f69cef1656c982cd3\": container with ID starting with ae03ee2f2bb5b3f2b2e0fb7b4f35755a8772ce14305d148f69cef1656c982cd3 not found: ID does not exist" containerID="ae03ee2f2bb5b3f2b2e0fb7b4f35755a8772ce14305d148f69cef1656c982cd3"
Sep 11 17:03:28.029734 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:28.029667    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae03ee2f2bb5b3f2b2e0fb7b4f35755a8772ce14305d148f69cef1656c982cd3"} err="failed to get container status \"ae03ee2f2bb5b3f2b2e0fb7b4f35755a8772ce14305d148f69cef1656c982cd3\": rpc error: code = NotFound desc = could not find container \"ae03ee2f2bb5b3f2b2e0fb7b4f35755a8772ce14305d148f69cef1656c982cd3\": container with ID starting with ae03ee2f2bb5b3f2b2e0fb7b4f35755a8772ce14305d148f69cef1656c982cd3 not found: ID does not exist"
Sep 11 17:03:28.030105 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:28.030089    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"]
Sep 11 17:03:28.040535 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:28.040508    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-j4rtp"]
Sep 11 17:03:29.321231 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:29.321198    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" path="/var/lib/kubelet/pods/73b13958-47c7-4dde-a8ff-d6728f1b7146/volumes"
Sep 11 17:03:34.985999 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.985956    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"]
Sep 11 17:03:34.986479 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.986328    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kube-rbac-proxy"
Sep 11 17:03:34.986479 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.986342    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kube-rbac-proxy"
Sep 11 17:03:34.986479 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.986353    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8493cf83-2eeb-486e-a6fe-5384b8d7ffae" containerName="kserve-container"
Sep 11 17:03:34.986479 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.986358    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8493cf83-2eeb-486e-a6fe-5384b8d7ffae" containerName="kserve-container"
Sep 11 17:03:34.986479 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.986368    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8493cf83-2eeb-486e-a6fe-5384b8d7ffae" containerName="kube-rbac-proxy"
Sep 11 17:03:34.986479 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.986374    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8493cf83-2eeb-486e-a6fe-5384b8d7ffae" containerName="kube-rbac-proxy"
Sep 11 17:03:34.986479 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.986386    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kserve-container"
Sep 11 17:03:34.986479 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.986391    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kserve-container"
Sep 11 17:03:34.986479 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.986401    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="storage-initializer"
Sep 11 17:03:34.986479 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.986408    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="storage-initializer"
Sep 11 17:03:34.986479 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.986413    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8493cf83-2eeb-486e-a6fe-5384b8d7ffae" containerName="storage-initializer"
Sep 11 17:03:34.986479 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.986420    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8493cf83-2eeb-486e-a6fe-5384b8d7ffae" containerName="storage-initializer"
Sep 11 17:03:34.986479 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.986474    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kserve-container"
Sep 11 17:03:34.986479 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.986482    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8493cf83-2eeb-486e-a6fe-5384b8d7ffae" containerName="kube-rbac-proxy"
Sep 11 17:03:34.986479 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.986489    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8493cf83-2eeb-486e-a6fe-5384b8d7ffae" containerName="kserve-container"
Sep 11 17:03:34.987264 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.986496    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="73b13958-47c7-4dde-a8ff-d6728f1b7146" containerName="kube-rbac-proxy"
Sep 11 17:03:34.991361 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.991338    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:34.996186 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.996147    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 11 17:03:34.996979 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.996949    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 11 17:03:34.997113 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.996949    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 17:03:34.997113 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.996951    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:03:34.997560 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.997537    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 11 17:03:34.997686 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.997624    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 17:03:34.997686 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:34.997632    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 17:03:35.006844 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.006819    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"]
Sep 11 17:03:35.111041 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.110995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ndxv\" (UniqueName: \"kubernetes.io/projected/9109550b-4f96-49c6-b067-e9decb636833-kube-api-access-2ndxv\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:35.111041 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.111058    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9109550b-4f96-49c6-b067-e9decb636833-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:35.111263 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.111080    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9109550b-4f96-49c6-b067-e9decb636833-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:35.111263 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.111133    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9109550b-4f96-49c6-b067-e9decb636833-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:35.111263 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.111167    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9109550b-4f96-49c6-b067-e9decb636833-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:35.212033 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.211991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9109550b-4f96-49c6-b067-e9decb636833-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:35.212033 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.212036    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9109550b-4f96-49c6-b067-e9decb636833-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:35.212339 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.212062    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9109550b-4f96-49c6-b067-e9decb636833-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:35.212339 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.212157    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ndxv\" (UniqueName: \"kubernetes.io/projected/9109550b-4f96-49c6-b067-e9decb636833-kube-api-access-2ndxv\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:35.212339 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.212229    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9109550b-4f96-49c6-b067-e9decb636833-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:35.212498 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.212479    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9109550b-4f96-49c6-b067-e9decb636833-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:35.214448 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.214430    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 11 17:03:35.214501 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.214457    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:03:35.214545 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.214432    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 11 17:03:35.220587 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.220572    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 17:03:35.223708 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.223680    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9109550b-4f96-49c6-b067-e9decb636833-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:35.223838 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.223680    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9109550b-4f96-49c6-b067-e9decb636833-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:35.225448 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.225426    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9109550b-4f96-49c6-b067-e9decb636833-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:35.231461 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.231441    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 17:03:35.241644 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.241581    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ndxv\" (UniqueName: \"kubernetes.io/projected/9109550b-4f96-49c6-b067-e9decb636833-kube-api-access-2ndxv\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:35.304452 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.304420    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 17:03:35.312819 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.312797    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:35.592223 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.592167    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"]
Sep 11 17:03:35.598576 ip-10-0-142-248 kubenswrapper[2561]: W0911 17:03:35.598528    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9109550b_4f96_49c6_b067_e9decb636833.slice/crio-b744f3761861ad45d60c206bfd777534d7f12a8b182af43a0e67ec3acb7b0a42 WatchSource:0}: Error finding container b744f3761861ad45d60c206bfd777534d7f12a8b182af43a0e67ec3acb7b0a42: Status 404 returned error can't find the container with id b744f3761861ad45d60c206bfd777534d7f12a8b182af43a0e67ec3acb7b0a42
Sep 11 17:03:35.602655 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:35.602634    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 11 17:03:36.045545 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:36.045499    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" event={"ID":"9109550b-4f96-49c6-b067-e9decb636833","Type":"ContainerStarted","Data":"3abfa815382b7567b79b083e97c217aa88dbea9dc2f2d6932cd142d364642fc4"}
Sep 11 17:03:36.045930 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:36.045589    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" event={"ID":"9109550b-4f96-49c6-b067-e9decb636833","Type":"ContainerStarted","Data":"b744f3761861ad45d60c206bfd777534d7f12a8b182af43a0e67ec3acb7b0a42"}
Sep 11 17:03:37.051989 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:37.051879    2561 generic.go:356] "Generic (PLEG): container finished" podID="9109550b-4f96-49c6-b067-e9decb636833" containerID="3abfa815382b7567b79b083e97c217aa88dbea9dc2f2d6932cd142d364642fc4" exitCode=0
Sep 11 17:03:37.051989 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:37.051970    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" event={"ID":"9109550b-4f96-49c6-b067-e9decb636833","Type":"ContainerDied","Data":"3abfa815382b7567b79b083e97c217aa88dbea9dc2f2d6932cd142d364642fc4"}
Sep 11 17:03:38.058659 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:38.058619    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" event={"ID":"9109550b-4f96-49c6-b067-e9decb636833","Type":"ContainerStarted","Data":"c16813bca6ed468c20505a9b058a950d8c838ab6cd6925b19f52e85e6f9c5416"}
Sep 11 17:03:38.059013 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:38.058692    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" event={"ID":"9109550b-4f96-49c6-b067-e9decb636833","Type":"ContainerStarted","Data":"f86f938e9efed5deeb6a24da246b65d584e2e8e4554953445047f7c90ea7d8fd"}
Sep 11 17:03:38.059013 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:38.058726    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:38.079934 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:38.079881    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" podStartSLOduration=3.000253573 podStartE2EDuration="4.079867214s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.079613641s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:03:38.078684548 +0000 UTC m=+2823.418778593" watchObservedRunningTime="2026-09-11 17:03:38.079867214 +0000 UTC m=+2823.419961260"
Sep 11 17:03:39.063588 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:39.063548    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:39.064855 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:39.064823    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 11 17:03:40.068270 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:40.068225    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 11 17:03:45.073032 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:45.072997    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:03:45.073668 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:45.073641    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 11 17:03:55.074287 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:03:55.074248    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 11 17:04:05.073902 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:04:05.073857    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 11 17:04:15.073682 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:04:15.073641    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 11 17:04:25.073604 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:04:25.073555    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 11 17:04:35.074222 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:04:35.074183    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 11 17:04:45.074502 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:04:45.074461    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 11 17:04:55.074093 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:04:55.074063    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:05:04.934671 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:04.934638    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"]
Sep 11 17:05:04.935162 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:04.934890    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kserve-container" containerID="cri-o://f86f938e9efed5deeb6a24da246b65d584e2e8e4554953445047f7c90ea7d8fd" gracePeriod=30
Sep 11 17:05:04.935162 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:04.934933    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kube-rbac-proxy" containerID="cri-o://c16813bca6ed468c20505a9b058a950d8c838ab6cd6925b19f52e85e6f9c5416" gracePeriod=30
Sep 11 17:05:05.068809 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:05.068767    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 11 17:05:05.074226 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:05.074199    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 11 17:05:05.474822 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:05.474789    2561 generic.go:356] "Generic (PLEG): container finished" podID="9109550b-4f96-49c6-b067-e9decb636833" containerID="c16813bca6ed468c20505a9b058a950d8c838ab6cd6925b19f52e85e6f9c5416" exitCode=2
Sep 11 17:05:05.475009 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:05.474861    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" event={"ID":"9109550b-4f96-49c6-b067-e9decb636833","Type":"ContainerDied","Data":"c16813bca6ed468c20505a9b058a950d8c838ab6cd6925b19f52e85e6f9c5416"}
Sep 11 17:05:06.080861 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.080823    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"]
Sep 11 17:05:06.084463 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.084443    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:06.086344 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.086322    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 11 17:05:06.086516 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.086498    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 11 17:05:06.096266 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.096243    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"]
Sep 11 17:05:06.195109 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.195076    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:06.195109 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.195113    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d57bv\" (UniqueName: \"kubernetes.io/projected/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kube-api-access-d57bv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:06.195394 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.195133    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:06.195394 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.195156    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:06.295847 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.295810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:06.295847 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.295846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d57bv\" (UniqueName: \"kubernetes.io/projected/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kube-api-access-d57bv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:06.296094 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.295866    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:06.296094 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.295892    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:06.296094 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:05:06.295978    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 11 17:05:06.296094 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:05:06.296055    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-proxy-tls podName:82576e51-80e5-4cd5-ba6f-61cfaa3484e8 nodeName:}" failed. No retries permitted until 2026-09-11 17:05:06.796033834 +0000 UTC m=+2912.136127857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf" (UID: "82576e51-80e5-4cd5-ba6f-61cfaa3484e8") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 11 17:05:06.296351 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.296330    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:06.296603 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.296586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:06.308142 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.308119    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d57bv\" (UniqueName: \"kubernetes.io/projected/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kube-api-access-d57bv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:06.801062 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.801020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:06.803580 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.803554    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:06.994659 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:06.994619    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:07.272978 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:07.272951    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"]
Sep 11 17:05:07.275532 ip-10-0-142-248 kubenswrapper[2561]: W0911 17:05:07.275504    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82576e51_80e5_4cd5_ba6f_61cfaa3484e8.slice/crio-4f82c0984e5359beeb040a4932ee567ec83058f7f1fd9e7f10bb2343cf95549d WatchSource:0}: Error finding container 4f82c0984e5359beeb040a4932ee567ec83058f7f1fd9e7f10bb2343cf95549d: Status 404 returned error can't find the container with id 4f82c0984e5359beeb040a4932ee567ec83058f7f1fd9e7f10bb2343cf95549d
Sep 11 17:05:07.486641 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:07.486599    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf" event={"ID":"82576e51-80e5-4cd5-ba6f-61cfaa3484e8","Type":"ContainerStarted","Data":"3e49016809b614afab17e0868f5f29096e30c19d3df3cb69a7146c3e43300e20"}
Sep 11 17:05:07.486838 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:07.486672    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf" event={"ID":"82576e51-80e5-4cd5-ba6f-61cfaa3484e8","Type":"ContainerStarted","Data":"4f82c0984e5359beeb040a4932ee567ec83058f7f1fd9e7f10bb2343cf95549d"}
Sep 11 17:05:09.681698 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:09.681670    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:05:09.725980 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:09.725908    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9109550b-4f96-49c6-b067-e9decb636833-cabundle-cert\") pod \"9109550b-4f96-49c6-b067-e9decb636833\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") "
Sep 11 17:05:09.725980 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:09.725973    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ndxv\" (UniqueName: \"kubernetes.io/projected/9109550b-4f96-49c6-b067-e9decb636833-kube-api-access-2ndxv\") pod \"9109550b-4f96-49c6-b067-e9decb636833\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") "
Sep 11 17:05:09.726222 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:09.725992    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9109550b-4f96-49c6-b067-e9decb636833-kserve-provision-location\") pod \"9109550b-4f96-49c6-b067-e9decb636833\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") "
Sep 11 17:05:09.726222 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:09.726060    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9109550b-4f96-49c6-b067-e9decb636833-proxy-tls\") pod \"9109550b-4f96-49c6-b067-e9decb636833\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") "
Sep 11 17:05:09.726222 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:09.726082    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9109550b-4f96-49c6-b067-e9decb636833-kube-rbac-proxy-sar-config\") pod \"9109550b-4f96-49c6-b067-e9decb636833\" (UID: \"9109550b-4f96-49c6-b067-e9decb636833\") "
Sep 11 17:05:09.726392 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:09.726328    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9109550b-4f96-49c6-b067-e9decb636833-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9109550b-4f96-49c6-b067-e9decb636833" (UID: "9109550b-4f96-49c6-b067-e9decb636833"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:05:09.726476 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:09.726329    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9109550b-4f96-49c6-b067-e9decb636833-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9109550b-4f96-49c6-b067-e9decb636833" (UID: "9109550b-4f96-49c6-b067-e9decb636833"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:05:09.726544 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:09.726495    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9109550b-4f96-49c6-b067-e9decb636833-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9109550b-4f96-49c6-b067-e9decb636833" (UID: "9109550b-4f96-49c6-b067-e9decb636833"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:05:09.728161 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:09.728139    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9109550b-4f96-49c6-b067-e9decb636833-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9109550b-4f96-49c6-b067-e9decb636833" (UID: "9109550b-4f96-49c6-b067-e9decb636833"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:05:09.728244 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:09.728193    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9109550b-4f96-49c6-b067-e9decb636833-kube-api-access-2ndxv" (OuterVolumeSpecName: "kube-api-access-2ndxv") pod "9109550b-4f96-49c6-b067-e9decb636833" (UID: "9109550b-4f96-49c6-b067-e9decb636833"). InnerVolumeSpecName "kube-api-access-2ndxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:05:09.826668 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:09.826629    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ndxv\" (UniqueName: \"kubernetes.io/projected/9109550b-4f96-49c6-b067-e9decb636833-kube-api-access-2ndxv\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:05:09.826668 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:09.826661    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9109550b-4f96-49c6-b067-e9decb636833-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:05:09.826668 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:09.826672    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9109550b-4f96-49c6-b067-e9decb636833-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:05:09.826892 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:09.826682    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9109550b-4f96-49c6-b067-e9decb636833-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:05:09.826892 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:09.826692    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9109550b-4f96-49c6-b067-e9decb636833-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:05:10.503916 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:10.503879    2561 generic.go:356] "Generic (PLEG): container finished" podID="9109550b-4f96-49c6-b067-e9decb636833" containerID="f86f938e9efed5deeb6a24da246b65d584e2e8e4554953445047f7c90ea7d8fd" exitCode=0
Sep 11 17:05:10.504082 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:10.503940    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"
Sep 11 17:05:10.504082 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:10.503967    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" event={"ID":"9109550b-4f96-49c6-b067-e9decb636833","Type":"ContainerDied","Data":"f86f938e9efed5deeb6a24da246b65d584e2e8e4554953445047f7c90ea7d8fd"}
Sep 11 17:05:10.504082 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:10.504040    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f" event={"ID":"9109550b-4f96-49c6-b067-e9decb636833","Type":"ContainerDied","Data":"b744f3761861ad45d60c206bfd777534d7f12a8b182af43a0e67ec3acb7b0a42"}
Sep 11 17:05:10.504198 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:10.504081    2561 scope.go:117] "RemoveContainer" containerID="c16813bca6ed468c20505a9b058a950d8c838ab6cd6925b19f52e85e6f9c5416"
Sep 11 17:05:10.513214 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:10.513195    2561 scope.go:117] "RemoveContainer" containerID="f86f938e9efed5deeb6a24da246b65d584e2e8e4554953445047f7c90ea7d8fd"
Sep 11 17:05:10.520712 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:10.520696    2561 scope.go:117] "RemoveContainer" containerID="3abfa815382b7567b79b083e97c217aa88dbea9dc2f2d6932cd142d364642fc4"
Sep 11 17:05:10.528098 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:10.528047    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"]
Sep 11 17:05:10.528154 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:10.528138    2561 scope.go:117] "RemoveContainer" containerID="c16813bca6ed468c20505a9b058a950d8c838ab6cd6925b19f52e85e6f9c5416"
Sep 11 17:05:10.528428 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:05:10.528411    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c16813bca6ed468c20505a9b058a950d8c838ab6cd6925b19f52e85e6f9c5416\": container with ID starting with c16813bca6ed468c20505a9b058a950d8c838ab6cd6925b19f52e85e6f9c5416 not found: ID does not exist" containerID="c16813bca6ed468c20505a9b058a950d8c838ab6cd6925b19f52e85e6f9c5416"
Sep 11 17:05:10.528485 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:10.528435    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c16813bca6ed468c20505a9b058a950d8c838ab6cd6925b19f52e85e6f9c5416"} err="failed to get container status \"c16813bca6ed468c20505a9b058a950d8c838ab6cd6925b19f52e85e6f9c5416\": rpc error: code = NotFound desc = could not find container \"c16813bca6ed468c20505a9b058a950d8c838ab6cd6925b19f52e85e6f9c5416\": container with ID starting with c16813bca6ed468c20505a9b058a950d8c838ab6cd6925b19f52e85e6f9c5416 not found: ID does not exist"
Sep 11 17:05:10.528485 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:10.528453    2561 scope.go:117] "RemoveContainer" containerID="f86f938e9efed5deeb6a24da246b65d584e2e8e4554953445047f7c90ea7d8fd"
Sep 11 17:05:10.528711 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:05:10.528697    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86f938e9efed5deeb6a24da246b65d584e2e8e4554953445047f7c90ea7d8fd\": container with ID starting with f86f938e9efed5deeb6a24da246b65d584e2e8e4554953445047f7c90ea7d8fd not found: ID does not exist" containerID="f86f938e9efed5deeb6a24da246b65d584e2e8e4554953445047f7c90ea7d8fd"
Sep 11 17:05:10.528756 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:10.528714    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86f938e9efed5deeb6a24da246b65d584e2e8e4554953445047f7c90ea7d8fd"} err="failed to get container status \"f86f938e9efed5deeb6a24da246b65d584e2e8e4554953445047f7c90ea7d8fd\": rpc error: code = NotFound desc = could not find container \"f86f938e9efed5deeb6a24da246b65d584e2e8e4554953445047f7c90ea7d8fd\": container with ID starting with f86f938e9efed5deeb6a24da246b65d584e2e8e4554953445047f7c90ea7d8fd not found: ID does not exist"
Sep 11 17:05:10.528756 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:10.528726    2561 scope.go:117] "RemoveContainer" containerID="3abfa815382b7567b79b083e97c217aa88dbea9dc2f2d6932cd142d364642fc4"
Sep 11 17:05:10.528942 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:05:10.528926    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3abfa815382b7567b79b083e97c217aa88dbea9dc2f2d6932cd142d364642fc4\": container with ID starting with 3abfa815382b7567b79b083e97c217aa88dbea9dc2f2d6932cd142d364642fc4 not found: ID does not exist" containerID="3abfa815382b7567b79b083e97c217aa88dbea9dc2f2d6932cd142d364642fc4"
Sep 11 17:05:10.528981 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:10.528946    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3abfa815382b7567b79b083e97c217aa88dbea9dc2f2d6932cd142d364642fc4"} err="failed to get container status \"3abfa815382b7567b79b083e97c217aa88dbea9dc2f2d6932cd142d364642fc4\": rpc error: code = NotFound desc = could not find container \"3abfa815382b7567b79b083e97c217aa88dbea9dc2f2d6932cd142d364642fc4\": container with ID starting with 3abfa815382b7567b79b083e97c217aa88dbea9dc2f2d6932cd142d364642fc4 not found: ID does not exist"
Sep 11 17:05:10.532575 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:10.532554    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-l478f"]
Sep 11 17:05:11.321679 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:11.321585    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9109550b-4f96-49c6-b067-e9decb636833" path="/var/lib/kubelet/pods/9109550b-4f96-49c6-b067-e9decb636833/volumes"
Sep 11 17:05:11.512583 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:11.512555    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf_82576e51-80e5-4cd5-ba6f-61cfaa3484e8/storage-initializer/0.log"
Sep 11 17:05:11.512737 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:11.512589    2561 generic.go:356] "Generic (PLEG): container finished" podID="82576e51-80e5-4cd5-ba6f-61cfaa3484e8" containerID="3e49016809b614afab17e0868f5f29096e30c19d3df3cb69a7146c3e43300e20" exitCode=1
Sep 11 17:05:11.512737 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:11.512612    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf" event={"ID":"82576e51-80e5-4cd5-ba6f-61cfaa3484e8","Type":"ContainerDied","Data":"3e49016809b614afab17e0868f5f29096e30c19d3df3cb69a7146c3e43300e20"}
Sep 11 17:05:12.519632 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:12.519603    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf_82576e51-80e5-4cd5-ba6f-61cfaa3484e8/storage-initializer/0.log"
Sep 11 17:05:12.520012 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:12.519690    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf" event={"ID":"82576e51-80e5-4cd5-ba6f-61cfaa3484e8","Type":"ContainerStarted","Data":"4746c32b892578537ba96cb483940f7a38e270982a4bac79280283df7e1ea97f"}
Sep 11 17:05:16.024698 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.024658    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"]
Sep 11 17:05:16.025089 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.024910    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf" podUID="82576e51-80e5-4cd5-ba6f-61cfaa3484e8" containerName="storage-initializer" containerID="cri-o://4746c32b892578537ba96cb483940f7a38e270982a4bac79280283df7e1ea97f" gracePeriod=30
Sep 11 17:05:16.154546 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.154524    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf_82576e51-80e5-4cd5-ba6f-61cfaa3484e8/storage-initializer/1.log"
Sep 11 17:05:16.154933 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.154916    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf_82576e51-80e5-4cd5-ba6f-61cfaa3484e8/storage-initializer/0.log"
Sep 11 17:05:16.157291 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.157277    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:16.284080 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.283994    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-proxy-tls\") pod \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") "
Sep 11 17:05:16.284080 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.284068    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d57bv\" (UniqueName: \"kubernetes.io/projected/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kube-api-access-d57bv\") pod \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") "
Sep 11 17:05:16.284308 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.284089    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kube-rbac-proxy-sar-config\") pod \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") "
Sep 11 17:05:16.284308 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.284214    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kserve-provision-location\") pod \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\" (UID: \"82576e51-80e5-4cd5-ba6f-61cfaa3484e8\") "
Sep 11 17:05:16.284491 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.284468    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "82576e51-80e5-4cd5-ba6f-61cfaa3484e8" (UID: "82576e51-80e5-4cd5-ba6f-61cfaa3484e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:05:16.284535 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.284476    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "82576e51-80e5-4cd5-ba6f-61cfaa3484e8" (UID: "82576e51-80e5-4cd5-ba6f-61cfaa3484e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:05:16.286341 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.286304    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "82576e51-80e5-4cd5-ba6f-61cfaa3484e8" (UID: "82576e51-80e5-4cd5-ba6f-61cfaa3484e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:05:16.286341 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.286337    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kube-api-access-d57bv" (OuterVolumeSpecName: "kube-api-access-d57bv") pod "82576e51-80e5-4cd5-ba6f-61cfaa3484e8" (UID: "82576e51-80e5-4cd5-ba6f-61cfaa3484e8"). InnerVolumeSpecName "kube-api-access-d57bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:05:16.384917 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.384882    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:05:16.384917 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.384911    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d57bv\" (UniqueName: \"kubernetes.io/projected/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kube-api-access-d57bv\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:05:16.384917 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.384922    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:05:16.385114 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.384931    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82576e51-80e5-4cd5-ba6f-61cfaa3484e8-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:05:16.543421 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.543343    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf_82576e51-80e5-4cd5-ba6f-61cfaa3484e8/storage-initializer/1.log"
Sep 11 17:05:16.543756 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.543739    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf_82576e51-80e5-4cd5-ba6f-61cfaa3484e8/storage-initializer/0.log"
Sep 11 17:05:16.543813 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.543779    2561 generic.go:356] "Generic (PLEG): container finished" podID="82576e51-80e5-4cd5-ba6f-61cfaa3484e8" containerID="4746c32b892578537ba96cb483940f7a38e270982a4bac79280283df7e1ea97f" exitCode=1
Sep 11 17:05:16.543850 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.543830    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"
Sep 11 17:05:16.543885 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.543840    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf" event={"ID":"82576e51-80e5-4cd5-ba6f-61cfaa3484e8","Type":"ContainerDied","Data":"4746c32b892578537ba96cb483940f7a38e270982a4bac79280283df7e1ea97f"}
Sep 11 17:05:16.543917 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.543906    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf" event={"ID":"82576e51-80e5-4cd5-ba6f-61cfaa3484e8","Type":"ContainerDied","Data":"4f82c0984e5359beeb040a4932ee567ec83058f7f1fd9e7f10bb2343cf95549d"}
Sep 11 17:05:16.543948 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.543933    2561 scope.go:117] "RemoveContainer" containerID="4746c32b892578537ba96cb483940f7a38e270982a4bac79280283df7e1ea97f"
Sep 11 17:05:16.552889 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.552871    2561 scope.go:117] "RemoveContainer" containerID="3e49016809b614afab17e0868f5f29096e30c19d3df3cb69a7146c3e43300e20"
Sep 11 17:05:16.560504 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.560486    2561 scope.go:117] "RemoveContainer" containerID="4746c32b892578537ba96cb483940f7a38e270982a4bac79280283df7e1ea97f"
Sep 11 17:05:16.560762 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:05:16.560745    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4746c32b892578537ba96cb483940f7a38e270982a4bac79280283df7e1ea97f\": container with ID starting with 4746c32b892578537ba96cb483940f7a38e270982a4bac79280283df7e1ea97f not found: ID does not exist" containerID="4746c32b892578537ba96cb483940f7a38e270982a4bac79280283df7e1ea97f"
Sep 11 17:05:16.560816 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.560770    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4746c32b892578537ba96cb483940f7a38e270982a4bac79280283df7e1ea97f"} err="failed to get container status \"4746c32b892578537ba96cb483940f7a38e270982a4bac79280283df7e1ea97f\": rpc error: code = NotFound desc = could not find container \"4746c32b892578537ba96cb483940f7a38e270982a4bac79280283df7e1ea97f\": container with ID starting with 4746c32b892578537ba96cb483940f7a38e270982a4bac79280283df7e1ea97f not found: ID does not exist"
Sep 11 17:05:16.560816 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.560788    2561 scope.go:117] "RemoveContainer" containerID="3e49016809b614afab17e0868f5f29096e30c19d3df3cb69a7146c3e43300e20"
Sep 11 17:05:16.561034 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:05:16.561016    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e49016809b614afab17e0868f5f29096e30c19d3df3cb69a7146c3e43300e20\": container with ID starting with 3e49016809b614afab17e0868f5f29096e30c19d3df3cb69a7146c3e43300e20 not found: ID does not exist" containerID="3e49016809b614afab17e0868f5f29096e30c19d3df3cb69a7146c3e43300e20"
Sep 11 17:05:16.561086 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.561045    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e49016809b614afab17e0868f5f29096e30c19d3df3cb69a7146c3e43300e20"} err="failed to get container status \"3e49016809b614afab17e0868f5f29096e30c19d3df3cb69a7146c3e43300e20\": rpc error: code = NotFound desc = could not find container \"3e49016809b614afab17e0868f5f29096e30c19d3df3cb69a7146c3e43300e20\": container with ID starting with 3e49016809b614afab17e0868f5f29096e30c19d3df3cb69a7146c3e43300e20 not found: ID does not exist"
Sep 11 17:05:16.586878 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.586840    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"]
Sep 11 17:05:16.599048 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:16.599013    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-msqpf"]
Sep 11 17:05:17.321109 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:05:17.321062    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82576e51-80e5-4cd5-ba6f-61cfaa3484e8" path="/var/lib/kubelet/pods/82576e51-80e5-4cd5-ba6f-61cfaa3484e8/volumes"
Sep 11 17:06:35.409703 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:35.409672    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 17:06:35.523792 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:35.523762    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 17:06:48.307715 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.307670    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"]
Sep 11 17:06:48.308228 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.308162    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kube-rbac-proxy"
Sep 11 17:06:48.308228 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.308188    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kube-rbac-proxy"
Sep 11 17:06:48.308228 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.308215    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="storage-initializer"
Sep 11 17:06:48.308228 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.308224    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="storage-initializer"
Sep 11 17:06:48.308464 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.308234    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82576e51-80e5-4cd5-ba6f-61cfaa3484e8" containerName="storage-initializer"
Sep 11 17:06:48.308464 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.308240    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="82576e51-80e5-4cd5-ba6f-61cfaa3484e8" containerName="storage-initializer"
Sep 11 17:06:48.308464 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.308253    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kserve-container"
Sep 11 17:06:48.308464 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.308258    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kserve-container"
Sep 11 17:06:48.308464 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.308264    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82576e51-80e5-4cd5-ba6f-61cfaa3484e8" containerName="storage-initializer"
Sep 11 17:06:48.308464 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.308269    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="82576e51-80e5-4cd5-ba6f-61cfaa3484e8" containerName="storage-initializer"
Sep 11 17:06:48.308464 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.308329    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kserve-container"
Sep 11 17:06:48.308464 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.308339    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="82576e51-80e5-4cd5-ba6f-61cfaa3484e8" containerName="storage-initializer"
Sep 11 17:06:48.308464 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.308345    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="82576e51-80e5-4cd5-ba6f-61cfaa3484e8" containerName="storage-initializer"
Sep 11 17:06:48.308464 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.308351    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9109550b-4f96-49c6-b067-e9decb636833" containerName="kube-rbac-proxy"
Sep 11 17:06:48.311648 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.311623    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:48.313914 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.313879    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 11 17:06:48.314063 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.313965    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 17:06:48.314063 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.314026    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 17:06:48.314245 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.314091    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 11 17:06:48.314245 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.314138    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 11 17:06:48.314375 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.314318    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 17:06:48.323756 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.323721    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"]
Sep 11 17:06:48.381084 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.381036    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d866447-6a09-4288-beb1-dd68c361cf50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:48.381084 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.381082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d866447-6a09-4288-beb1-dd68c361cf50-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:48.381341 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.381110    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmz24\" (UniqueName: \"kubernetes.io/projected/3d866447-6a09-4288-beb1-dd68c361cf50-kube-api-access-bmz24\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:48.381341 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.381141    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d866447-6a09-4288-beb1-dd68c361cf50-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:48.482256 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.482207    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmz24\" (UniqueName: \"kubernetes.io/projected/3d866447-6a09-4288-beb1-dd68c361cf50-kube-api-access-bmz24\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:48.482473 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.482276    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d866447-6a09-4288-beb1-dd68c361cf50-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:48.482473 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.482384    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d866447-6a09-4288-beb1-dd68c361cf50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:48.482473 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.482418    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d866447-6a09-4288-beb1-dd68c361cf50-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:48.482473 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:06:48.482434    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 11 17:06:48.482692 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:06:48.482527    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d866447-6a09-4288-beb1-dd68c361cf50-proxy-tls podName:3d866447-6a09-4288-beb1-dd68c361cf50 nodeName:}" failed. No retries permitted until 2026-09-11 17:06:48.982504084 +0000 UTC m=+3014.322598112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3d866447-6a09-4288-beb1-dd68c361cf50-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl" (UID: "3d866447-6a09-4288-beb1-dd68c361cf50") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 11 17:06:48.482886 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.482865    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d866447-6a09-4288-beb1-dd68c361cf50-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:48.483168 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.483146    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d866447-6a09-4288-beb1-dd68c361cf50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:48.491048 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.491008    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmz24\" (UniqueName: \"kubernetes.io/projected/3d866447-6a09-4288-beb1-dd68c361cf50-kube-api-access-bmz24\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:48.987781 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.987715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d866447-6a09-4288-beb1-dd68c361cf50-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:48.990285 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:48.990260    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d866447-6a09-4288-beb1-dd68c361cf50-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:49.223707 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:49.223657    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:49.500504 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:49.500474    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"]
Sep 11 17:06:49.503068 ip-10-0-142-248 kubenswrapper[2561]: W0911 17:06:49.503038    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d866447_6a09_4288_beb1_dd68c361cf50.slice/crio-fd9bb80e1a1e09b3ed1641be7ccfca7cb30ec4a4123a0d6e8007b6ec965f7302 WatchSource:0}: Error finding container fd9bb80e1a1e09b3ed1641be7ccfca7cb30ec4a4123a0d6e8007b6ec965f7302: Status 404 returned error can't find the container with id fd9bb80e1a1e09b3ed1641be7ccfca7cb30ec4a4123a0d6e8007b6ec965f7302
Sep 11 17:06:49.964007 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:49.963966    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl" event={"ID":"3d866447-6a09-4288-beb1-dd68c361cf50","Type":"ContainerStarted","Data":"211d00b1d36c75e6d55eeb50bca66cd251acc6cf7f622eca7686924fef871c12"}
Sep 11 17:06:49.964232 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:49.964029    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl" event={"ID":"3d866447-6a09-4288-beb1-dd68c361cf50","Type":"ContainerStarted","Data":"fd9bb80e1a1e09b3ed1641be7ccfca7cb30ec4a4123a0d6e8007b6ec965f7302"}
Sep 11 17:06:53.986927 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:53.986895    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl_3d866447-6a09-4288-beb1-dd68c361cf50/storage-initializer/0.log"
Sep 11 17:06:53.987317 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:53.986938    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d866447-6a09-4288-beb1-dd68c361cf50" containerID="211d00b1d36c75e6d55eeb50bca66cd251acc6cf7f622eca7686924fef871c12" exitCode=1
Sep 11 17:06:53.987317 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:53.987014    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl" event={"ID":"3d866447-6a09-4288-beb1-dd68c361cf50","Type":"ContainerDied","Data":"211d00b1d36c75e6d55eeb50bca66cd251acc6cf7f622eca7686924fef871c12"}
Sep 11 17:06:54.999662 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:54.999634    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl_3d866447-6a09-4288-beb1-dd68c361cf50/storage-initializer/0.log"
Sep 11 17:06:55.000046 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:54.999751    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl" event={"ID":"3d866447-6a09-4288-beb1-dd68c361cf50","Type":"ContainerStarted","Data":"0aec65cfa9bae38129c87e8085944020b015b87123084e027c5be23deaf5b75e"}
Sep 11 17:06:58.315682 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.315648    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"]
Sep 11 17:06:58.316090 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.315884    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl" podUID="3d866447-6a09-4288-beb1-dd68c361cf50" containerName="storage-initializer" containerID="cri-o://0aec65cfa9bae38129c87e8085944020b015b87123084e027c5be23deaf5b75e" gracePeriod=30
Sep 11 17:06:58.557157 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.557132    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl_3d866447-6a09-4288-beb1-dd68c361cf50/storage-initializer/1.log"
Sep 11 17:06:58.557605 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.557590    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl_3d866447-6a09-4288-beb1-dd68c361cf50/storage-initializer/0.log"
Sep 11 17:06:58.559824 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.559809    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:58.672880 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.672844    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmz24\" (UniqueName: \"kubernetes.io/projected/3d866447-6a09-4288-beb1-dd68c361cf50-kube-api-access-bmz24\") pod \"3d866447-6a09-4288-beb1-dd68c361cf50\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") "
Sep 11 17:06:58.673026 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.672895    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d866447-6a09-4288-beb1-dd68c361cf50-kube-rbac-proxy-sar-config\") pod \"3d866447-6a09-4288-beb1-dd68c361cf50\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") "
Sep 11 17:06:58.673026 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.672928    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d866447-6a09-4288-beb1-dd68c361cf50-proxy-tls\") pod \"3d866447-6a09-4288-beb1-dd68c361cf50\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") "
Sep 11 17:06:58.673026 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.672971    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d866447-6a09-4288-beb1-dd68c361cf50-kserve-provision-location\") pod \"3d866447-6a09-4288-beb1-dd68c361cf50\" (UID: \"3d866447-6a09-4288-beb1-dd68c361cf50\") "
Sep 11 17:06:58.673371 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.673338    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d866447-6a09-4288-beb1-dd68c361cf50-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d866447-6a09-4288-beb1-dd68c361cf50" (UID: "3d866447-6a09-4288-beb1-dd68c361cf50"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:06:58.673371 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.673356    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d866447-6a09-4288-beb1-dd68c361cf50-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3d866447-6a09-4288-beb1-dd68c361cf50" (UID: "3d866447-6a09-4288-beb1-dd68c361cf50"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:06:58.675107 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.675086    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d866447-6a09-4288-beb1-dd68c361cf50-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d866447-6a09-4288-beb1-dd68c361cf50" (UID: "3d866447-6a09-4288-beb1-dd68c361cf50"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:06:58.675227 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.675212    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d866447-6a09-4288-beb1-dd68c361cf50-kube-api-access-bmz24" (OuterVolumeSpecName: "kube-api-access-bmz24") pod "3d866447-6a09-4288-beb1-dd68c361cf50" (UID: "3d866447-6a09-4288-beb1-dd68c361cf50"). InnerVolumeSpecName "kube-api-access-bmz24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:06:58.774389 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.774349    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d866447-6a09-4288-beb1-dd68c361cf50-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:06:58.774389 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.774384    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d866447-6a09-4288-beb1-dd68c361cf50-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:06:58.774389 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.774394    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmz24\" (UniqueName: \"kubernetes.io/projected/3d866447-6a09-4288-beb1-dd68c361cf50-kube-api-access-bmz24\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:06:58.774623 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:58.774406    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d866447-6a09-4288-beb1-dd68c361cf50-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:06:59.022778 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:59.022701    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl_3d866447-6a09-4288-beb1-dd68c361cf50/storage-initializer/1.log"
Sep 11 17:06:59.023092 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:59.023078    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl_3d866447-6a09-4288-beb1-dd68c361cf50/storage-initializer/0.log"
Sep 11 17:06:59.023144 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:59.023114    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d866447-6a09-4288-beb1-dd68c361cf50" containerID="0aec65cfa9bae38129c87e8085944020b015b87123084e027c5be23deaf5b75e" exitCode=1
Sep 11 17:06:59.023218 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:59.023204    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"
Sep 11 17:06:59.023255 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:59.023208    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl" event={"ID":"3d866447-6a09-4288-beb1-dd68c361cf50","Type":"ContainerDied","Data":"0aec65cfa9bae38129c87e8085944020b015b87123084e027c5be23deaf5b75e"}
Sep 11 17:06:59.023302 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:59.023284    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl" event={"ID":"3d866447-6a09-4288-beb1-dd68c361cf50","Type":"ContainerDied","Data":"fd9bb80e1a1e09b3ed1641be7ccfca7cb30ec4a4123a0d6e8007b6ec965f7302"}
Sep 11 17:06:59.023340 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:59.023312    2561 scope.go:117] "RemoveContainer" containerID="0aec65cfa9bae38129c87e8085944020b015b87123084e027c5be23deaf5b75e"
Sep 11 17:06:59.032983 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:59.032962    2561 scope.go:117] "RemoveContainer" containerID="211d00b1d36c75e6d55eeb50bca66cd251acc6cf7f622eca7686924fef871c12"
Sep 11 17:06:59.041057 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:59.041026    2561 scope.go:117] "RemoveContainer" containerID="0aec65cfa9bae38129c87e8085944020b015b87123084e027c5be23deaf5b75e"
Sep 11 17:06:59.041367 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:06:59.041345    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aec65cfa9bae38129c87e8085944020b015b87123084e027c5be23deaf5b75e\": container with ID starting with 0aec65cfa9bae38129c87e8085944020b015b87123084e027c5be23deaf5b75e not found: ID does not exist" containerID="0aec65cfa9bae38129c87e8085944020b015b87123084e027c5be23deaf5b75e"
Sep 11 17:06:59.041465 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:59.041380    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aec65cfa9bae38129c87e8085944020b015b87123084e027c5be23deaf5b75e"} err="failed to get container status \"0aec65cfa9bae38129c87e8085944020b015b87123084e027c5be23deaf5b75e\": rpc error: code = NotFound desc = could not find container \"0aec65cfa9bae38129c87e8085944020b015b87123084e027c5be23deaf5b75e\": container with ID starting with 0aec65cfa9bae38129c87e8085944020b015b87123084e027c5be23deaf5b75e not found: ID does not exist"
Sep 11 17:06:59.041465 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:59.041403    2561 scope.go:117] "RemoveContainer" containerID="211d00b1d36c75e6d55eeb50bca66cd251acc6cf7f622eca7686924fef871c12"
Sep 11 17:06:59.041707 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:06:59.041689    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211d00b1d36c75e6d55eeb50bca66cd251acc6cf7f622eca7686924fef871c12\": container with ID starting with 211d00b1d36c75e6d55eeb50bca66cd251acc6cf7f622eca7686924fef871c12 not found: ID does not exist" containerID="211d00b1d36c75e6d55eeb50bca66cd251acc6cf7f622eca7686924fef871c12"
Sep 11 17:06:59.041742 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:59.041713    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211d00b1d36c75e6d55eeb50bca66cd251acc6cf7f622eca7686924fef871c12"} err="failed to get container status \"211d00b1d36c75e6d55eeb50bca66cd251acc6cf7f622eca7686924fef871c12\": rpc error: code = NotFound desc = could not find container \"211d00b1d36c75e6d55eeb50bca66cd251acc6cf7f622eca7686924fef871c12\": container with ID starting with 211d00b1d36c75e6d55eeb50bca66cd251acc6cf7f622eca7686924fef871c12 not found: ID does not exist"
Sep 11 17:06:59.065657 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:59.065626    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"]
Sep 11 17:06:59.071947 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:59.071917    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-82djl"]
Sep 11 17:06:59.321513 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:06:59.321435    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d866447-6a09-4288-beb1-dd68c361cf50" path="/var/lib/kubelet/pods/3d866447-6a09-4288-beb1-dd68c361cf50/volumes"
Sep 11 17:08:16.427273 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.427230    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"]
Sep 11 17:08:16.427808 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.427583    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d866447-6a09-4288-beb1-dd68c361cf50" containerName="storage-initializer"
Sep 11 17:08:16.427808 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.427595    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d866447-6a09-4288-beb1-dd68c361cf50" containerName="storage-initializer"
Sep 11 17:08:16.427808 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.427618    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d866447-6a09-4288-beb1-dd68c361cf50" containerName="storage-initializer"
Sep 11 17:08:16.427808 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.427624    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d866447-6a09-4288-beb1-dd68c361cf50" containerName="storage-initializer"
Sep 11 17:08:16.427808 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.427670    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d866447-6a09-4288-beb1-dd68c361cf50" containerName="storage-initializer"
Sep 11 17:08:16.427808 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.427778    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d866447-6a09-4288-beb1-dd68c361cf50" containerName="storage-initializer"
Sep 11 17:08:16.430887 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.430868    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:16.433456 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.433420    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 17:08:16.433754 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.433728    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:08:16.433891 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.433783    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 17:08:16.433891 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.433832    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zr6pv\""
Sep 11 17:08:16.434024 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.433959    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 11 17:08:16.434092 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.434074    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 11 17:08:16.434159 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.434149    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 17:08:16.441206 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.441165    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"]
Sep 11 17:08:16.462537 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.462502    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-bnjbw\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:16.462537 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.462536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19b546c1-e7fa-4d0a-8241-92d5304be4b3-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-bnjbw\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:16.462806 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.462565    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-bnjbw\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:16.462806 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.462662    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19b546c1-e7fa-4d0a-8241-92d5304be4b3-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-bnjbw\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:16.462806 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.462699    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njnxc\" (UniqueName: \"kubernetes.io/projected/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kube-api-access-njnxc\") pod \"isvc-pmml-predictor-8674795f59-bnjbw\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:16.563487 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.563450    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-bnjbw\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:16.563487 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.563489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19b546c1-e7fa-4d0a-8241-92d5304be4b3-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-bnjbw\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:16.563759 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.563510    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-bnjbw\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:16.563759 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.563566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19b546c1-e7fa-4d0a-8241-92d5304be4b3-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-bnjbw\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:16.563759 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.563597    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njnxc\" (UniqueName: \"kubernetes.io/projected/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kube-api-access-njnxc\") pod \"isvc-pmml-predictor-8674795f59-bnjbw\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:16.563948 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:08:16.563894    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Sep 11 17:08:16.564006 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:08:16.563970    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b546c1-e7fa-4d0a-8241-92d5304be4b3-proxy-tls podName:19b546c1-e7fa-4d0a-8241-92d5304be4b3 nodeName:}" failed. No retries permitted until 2026-09-11 17:08:17.063950109 +0000 UTC m=+3102.404044138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/19b546c1-e7fa-4d0a-8241-92d5304be4b3-proxy-tls") pod "isvc-pmml-predictor-8674795f59-bnjbw" (UID: "19b546c1-e7fa-4d0a-8241-92d5304be4b3") : secret "isvc-pmml-predictor-serving-cert" not found
Sep 11 17:08:16.564302 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.564272    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-bnjbw\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:16.564456 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.564311    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-bnjbw\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:16.564554 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.564535    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19b546c1-e7fa-4d0a-8241-92d5304be4b3-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-bnjbw\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:16.575805 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:16.575776    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njnxc\" (UniqueName: \"kubernetes.io/projected/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kube-api-access-njnxc\") pod \"isvc-pmml-predictor-8674795f59-bnjbw\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:17.066894 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:17.066854    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19b546c1-e7fa-4d0a-8241-92d5304be4b3-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-bnjbw\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:17.069489 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:17.069467    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19b546c1-e7fa-4d0a-8241-92d5304be4b3-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-bnjbw\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:17.343905 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:17.343811    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:17.622986 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:17.622899    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"]
Sep 11 17:08:17.626069 ip-10-0-142-248 kubenswrapper[2561]: W0911 17:08:17.626041    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b546c1_e7fa_4d0a_8241_92d5304be4b3.slice/crio-555c51af2e2eda35e8811a70f22dc1bb9be0ac75e12e4782ff72493951e20c9d WatchSource:0}: Error finding container 555c51af2e2eda35e8811a70f22dc1bb9be0ac75e12e4782ff72493951e20c9d: Status 404 returned error can't find the container with id 555c51af2e2eda35e8811a70f22dc1bb9be0ac75e12e4782ff72493951e20c9d
Sep 11 17:08:18.398063 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:18.398026    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" event={"ID":"19b546c1-e7fa-4d0a-8241-92d5304be4b3","Type":"ContainerStarted","Data":"0dfc6df174dc02a2e00e7b7423181cb242a346b62070bea31148509e62136127"}
Sep 11 17:08:18.398282 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:18.398122    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" event={"ID":"19b546c1-e7fa-4d0a-8241-92d5304be4b3","Type":"ContainerStarted","Data":"555c51af2e2eda35e8811a70f22dc1bb9be0ac75e12e4782ff72493951e20c9d"}
Sep 11 17:08:22.418856 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:22.418819    2561 generic.go:356] "Generic (PLEG): container finished" podID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerID="0dfc6df174dc02a2e00e7b7423181cb242a346b62070bea31148509e62136127" exitCode=0
Sep 11 17:08:22.419254 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:22.418876    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" event={"ID":"19b546c1-e7fa-4d0a-8241-92d5304be4b3","Type":"ContainerDied","Data":"0dfc6df174dc02a2e00e7b7423181cb242a346b62070bea31148509e62136127"}
Sep 11 17:08:22.420231 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:22.420207    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:08:30.480427 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:30.480392    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" event={"ID":"19b546c1-e7fa-4d0a-8241-92d5304be4b3","Type":"ContainerStarted","Data":"cfbf9bd6d8a25e84bcc90b2c0f79196a6777cce69d86bea28c4704caf31a899b"}
Sep 11 17:08:30.480886 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:30.480459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" event={"ID":"19b546c1-e7fa-4d0a-8241-92d5304be4b3","Type":"ContainerStarted","Data":"f8feccb5ef480a59ee7952eac1140753688e70cf0bebeabcb56d1f7ad5ecbaf9"}
Sep 11 17:08:30.480886 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:30.480658    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:30.501905 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:30.501850    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" podStartSLOduration=3.5567110509999997 podStartE2EDuration="14.50183378s" totalImagesPullingTime="7.0922686s" totalInitContainerRuntime="3.852854129s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:08:30.499529398 +0000 UTC m=+3115.839623444" watchObservedRunningTime="2026-09-11 17:08:30.50183378 +0000 UTC m=+3115.841927825"
Sep 11 17:08:31.485930 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:31.485895    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:31.487049 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:31.487020    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 11 17:08:32.490685 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:32.490645    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 11 17:08:37.495153 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:37.495122    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:08:37.495788 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:37.495763    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 11 17:08:47.496706 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:47.496666    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 11 17:08:57.495856 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:08:57.495774    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 11 17:09:07.496284 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:09:07.496237    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 11 17:09:17.496818 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:09:17.496775    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 11 17:09:27.496095 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:09:27.496036    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 11 17:09:37.495896 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:09:37.495852    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 11 17:09:42.316268 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:09:42.316220    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 11 17:09:52.316875 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:09:52.316842    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:09:57.492855 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:09:57.492822    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"]
Sep 11 17:09:57.493324 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:09:57.493099    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kserve-container" containerID="cri-o://f8feccb5ef480a59ee7952eac1140753688e70cf0bebeabcb56d1f7ad5ecbaf9" gracePeriod=30
Sep 11 17:09:57.493324 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:09:57.493129    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kube-rbac-proxy" containerID="cri-o://cfbf9bd6d8a25e84bcc90b2c0f79196a6777cce69d86bea28c4704caf31a899b" gracePeriod=30
Sep 11 17:09:57.900845 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:09:57.900805    2561 generic.go:356] "Generic (PLEG): container finished" podID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerID="cfbf9bd6d8a25e84bcc90b2c0f79196a6777cce69d86bea28c4704caf31a899b" exitCode=2
Sep 11 17:09:57.901001 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:09:57.900875    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" event={"ID":"19b546c1-e7fa-4d0a-8241-92d5304be4b3","Type":"ContainerDied","Data":"cfbf9bd6d8a25e84bcc90b2c0f79196a6777cce69d86bea28c4704caf31a899b"}
Sep 11 17:10:01.139497 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.139467    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:10:01.172879 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.172845    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kserve-provision-location\") pod \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") "
Sep 11 17:10:01.173043 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.172894    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19b546c1-e7fa-4d0a-8241-92d5304be4b3-cabundle-cert\") pod \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") "
Sep 11 17:10:01.173043 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.172930    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njnxc\" (UniqueName: \"kubernetes.io/projected/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kube-api-access-njnxc\") pod \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") "
Sep 11 17:10:01.173043 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.172968    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kube-rbac-proxy-sar-config\") pod \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") "
Sep 11 17:10:01.173043 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.173006    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19b546c1-e7fa-4d0a-8241-92d5304be4b3-proxy-tls\") pod \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\" (UID: \"19b546c1-e7fa-4d0a-8241-92d5304be4b3\") "
Sep 11 17:10:01.173275 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.173156    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19b546c1-e7fa-4d0a-8241-92d5304be4b3" (UID: "19b546c1-e7fa-4d0a-8241-92d5304be4b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:10:01.173337 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.173300    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b546c1-e7fa-4d0a-8241-92d5304be4b3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "19b546c1-e7fa-4d0a-8241-92d5304be4b3" (UID: "19b546c1-e7fa-4d0a-8241-92d5304be4b3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:10:01.173393 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.173353    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kserve-provision-location\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:10:01.173441 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.173407    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "19b546c1-e7fa-4d0a-8241-92d5304be4b3" (UID: "19b546c1-e7fa-4d0a-8241-92d5304be4b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:10:01.175217 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.175166    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b546c1-e7fa-4d0a-8241-92d5304be4b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19b546c1-e7fa-4d0a-8241-92d5304be4b3" (UID: "19b546c1-e7fa-4d0a-8241-92d5304be4b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:10:01.175343 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.175322    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kube-api-access-njnxc" (OuterVolumeSpecName: "kube-api-access-njnxc") pod "19b546c1-e7fa-4d0a-8241-92d5304be4b3" (UID: "19b546c1-e7fa-4d0a-8241-92d5304be4b3"). InnerVolumeSpecName "kube-api-access-njnxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:10:01.274548 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.274447    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19b546c1-e7fa-4d0a-8241-92d5304be4b3-proxy-tls\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:10:01.274548 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.274490    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19b546c1-e7fa-4d0a-8241-92d5304be4b3-cabundle-cert\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:10:01.274548 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.274500    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njnxc\" (UniqueName: \"kubernetes.io/projected/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kube-api-access-njnxc\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:10:01.274548 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.274509    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19b546c1-e7fa-4d0a-8241-92d5304be4b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:10:01.920776 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.920736    2561 generic.go:356] "Generic (PLEG): container finished" podID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerID="f8feccb5ef480a59ee7952eac1140753688e70cf0bebeabcb56d1f7ad5ecbaf9" exitCode=0
Sep 11 17:10:01.920950 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.920784    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"
Sep 11 17:10:01.920950 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.920782    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" event={"ID":"19b546c1-e7fa-4d0a-8241-92d5304be4b3","Type":"ContainerDied","Data":"f8feccb5ef480a59ee7952eac1140753688e70cf0bebeabcb56d1f7ad5ecbaf9"}
Sep 11 17:10:01.920950 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.920854    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw" event={"ID":"19b546c1-e7fa-4d0a-8241-92d5304be4b3","Type":"ContainerDied","Data":"555c51af2e2eda35e8811a70f22dc1bb9be0ac75e12e4782ff72493951e20c9d"}
Sep 11 17:10:01.920950 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.920886    2561 scope.go:117] "RemoveContainer" containerID="cfbf9bd6d8a25e84bcc90b2c0f79196a6777cce69d86bea28c4704caf31a899b"
Sep 11 17:10:01.929389 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.929372    2561 scope.go:117] "RemoveContainer" containerID="f8feccb5ef480a59ee7952eac1140753688e70cf0bebeabcb56d1f7ad5ecbaf9"
Sep 11 17:10:01.937099 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.937069    2561 scope.go:117] "RemoveContainer" containerID="0dfc6df174dc02a2e00e7b7423181cb242a346b62070bea31148509e62136127"
Sep 11 17:10:01.944722 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.944701    2561 scope.go:117] "RemoveContainer" containerID="cfbf9bd6d8a25e84bcc90b2c0f79196a6777cce69d86bea28c4704caf31a899b"
Sep 11 17:10:01.945012 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:10:01.944993    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfbf9bd6d8a25e84bcc90b2c0f79196a6777cce69d86bea28c4704caf31a899b\": container with ID starting with cfbf9bd6d8a25e84bcc90b2c0f79196a6777cce69d86bea28c4704caf31a899b not found: ID does not exist" containerID="cfbf9bd6d8a25e84bcc90b2c0f79196a6777cce69d86bea28c4704caf31a899b"
Sep 11 17:10:01.945059 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.945024    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfbf9bd6d8a25e84bcc90b2c0f79196a6777cce69d86bea28c4704caf31a899b"} err="failed to get container status \"cfbf9bd6d8a25e84bcc90b2c0f79196a6777cce69d86bea28c4704caf31a899b\": rpc error: code = NotFound desc = could not find container \"cfbf9bd6d8a25e84bcc90b2c0f79196a6777cce69d86bea28c4704caf31a899b\": container with ID starting with cfbf9bd6d8a25e84bcc90b2c0f79196a6777cce69d86bea28c4704caf31a899b not found: ID does not exist"
Sep 11 17:10:01.945059 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.945053    2561 scope.go:117] "RemoveContainer" containerID="f8feccb5ef480a59ee7952eac1140753688e70cf0bebeabcb56d1f7ad5ecbaf9"
Sep 11 17:10:01.945334 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:10:01.945316    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8feccb5ef480a59ee7952eac1140753688e70cf0bebeabcb56d1f7ad5ecbaf9\": container with ID starting with f8feccb5ef480a59ee7952eac1140753688e70cf0bebeabcb56d1f7ad5ecbaf9 not found: ID does not exist" containerID="f8feccb5ef480a59ee7952eac1140753688e70cf0bebeabcb56d1f7ad5ecbaf9"
Sep 11 17:10:01.945401 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.945338    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8feccb5ef480a59ee7952eac1140753688e70cf0bebeabcb56d1f7ad5ecbaf9"} err="failed to get container status \"f8feccb5ef480a59ee7952eac1140753688e70cf0bebeabcb56d1f7ad5ecbaf9\": rpc error: code = NotFound desc = could not find container \"f8feccb5ef480a59ee7952eac1140753688e70cf0bebeabcb56d1f7ad5ecbaf9\": container with ID starting with f8feccb5ef480a59ee7952eac1140753688e70cf0bebeabcb56d1f7ad5ecbaf9 not found: ID does not exist"
Sep 11 17:10:01.945401 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.945350    2561 scope.go:117] "RemoveContainer" containerID="0dfc6df174dc02a2e00e7b7423181cb242a346b62070bea31148509e62136127"
Sep 11 17:10:01.945595 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:10:01.945576    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dfc6df174dc02a2e00e7b7423181cb242a346b62070bea31148509e62136127\": container with ID starting with 0dfc6df174dc02a2e00e7b7423181cb242a346b62070bea31148509e62136127 not found: ID does not exist" containerID="0dfc6df174dc02a2e00e7b7423181cb242a346b62070bea31148509e62136127"
Sep 11 17:10:01.945635 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.945601    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dfc6df174dc02a2e00e7b7423181cb242a346b62070bea31148509e62136127"} err="failed to get container status \"0dfc6df174dc02a2e00e7b7423181cb242a346b62070bea31148509e62136127\": rpc error: code = NotFound desc = could not find container \"0dfc6df174dc02a2e00e7b7423181cb242a346b62070bea31148509e62136127\": container with ID starting with 0dfc6df174dc02a2e00e7b7423181cb242a346b62070bea31148509e62136127 not found: ID does not exist"
Sep 11 17:10:01.951923 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.951900    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"]
Sep 11 17:10:01.961771 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:01.961747    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-bnjbw"]
Sep 11 17:10:03.321331 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:10:03.321297    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" path="/var/lib/kubelet/pods/19b546c1-e7fa-4d0a-8241-92d5304be4b3/volumes"
Sep 11 17:11:35.441208 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:11:35.441168    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 17:11:35.559774 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:11:35.559746    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 17:12:51.204764 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.204716    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2zgmz/must-gather-dmjxt"]
Sep 11 17:12:51.205319 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.205241    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="storage-initializer"
Sep 11 17:12:51.205319 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.205260    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="storage-initializer"
Sep 11 17:12:51.205319 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.205271    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kube-rbac-proxy"
Sep 11 17:12:51.205319 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.205277    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kube-rbac-proxy"
Sep 11 17:12:51.205319 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.205289    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kserve-container"
Sep 11 17:12:51.205319 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.205295    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kserve-container"
Sep 11 17:12:51.205621 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.205365    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kube-rbac-proxy"
Sep 11 17:12:51.205621 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.205374    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="19b546c1-e7fa-4d0a-8241-92d5304be4b3" containerName="kserve-container"
Sep 11 17:12:51.208431 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.208414    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2zgmz/must-gather-dmjxt"
Sep 11 17:12:51.210296 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.210271    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2zgmz\"/\"kube-root-ca.crt\""
Sep 11 17:12:51.210296 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.210287    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2zgmz\"/\"openshift-service-ca.crt\""
Sep 11 17:12:51.215069 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.215042    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2zgmz/must-gather-dmjxt"]
Sep 11 17:12:51.257426 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.257397    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/74c43b2d-c090-4a00-bc31-0c8783bc0633-must-gather-output\") pod \"must-gather-dmjxt\" (UID: \"74c43b2d-c090-4a00-bc31-0c8783bc0633\") " pod="openshift-must-gather-2zgmz/must-gather-dmjxt"
Sep 11 17:12:51.257593 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.257437    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2qc\" (UniqueName: \"kubernetes.io/projected/74c43b2d-c090-4a00-bc31-0c8783bc0633-kube-api-access-lf2qc\") pod \"must-gather-dmjxt\" (UID: \"74c43b2d-c090-4a00-bc31-0c8783bc0633\") " pod="openshift-must-gather-2zgmz/must-gather-dmjxt"
Sep 11 17:12:51.358838 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.358804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/74c43b2d-c090-4a00-bc31-0c8783bc0633-must-gather-output\") pod \"must-gather-dmjxt\" (UID: \"74c43b2d-c090-4a00-bc31-0c8783bc0633\") " pod="openshift-must-gather-2zgmz/must-gather-dmjxt"
Sep 11 17:12:51.358838 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.358845    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2qc\" (UniqueName: \"kubernetes.io/projected/74c43b2d-c090-4a00-bc31-0c8783bc0633-kube-api-access-lf2qc\") pod \"must-gather-dmjxt\" (UID: \"74c43b2d-c090-4a00-bc31-0c8783bc0633\") " pod="openshift-must-gather-2zgmz/must-gather-dmjxt"
Sep 11 17:12:51.359208 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.359161    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/74c43b2d-c090-4a00-bc31-0c8783bc0633-must-gather-output\") pod \"must-gather-dmjxt\" (UID: \"74c43b2d-c090-4a00-bc31-0c8783bc0633\") " pod="openshift-must-gather-2zgmz/must-gather-dmjxt"
Sep 11 17:12:51.367273 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.367249    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2qc\" (UniqueName: \"kubernetes.io/projected/74c43b2d-c090-4a00-bc31-0c8783bc0633-kube-api-access-lf2qc\") pod \"must-gather-dmjxt\" (UID: \"74c43b2d-c090-4a00-bc31-0c8783bc0633\") " pod="openshift-must-gather-2zgmz/must-gather-dmjxt"
Sep 11 17:12:51.527011 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.526928    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2zgmz/must-gather-dmjxt"
Sep 11 17:12:51.798233 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:51.798206    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2zgmz/must-gather-dmjxt"]
Sep 11 17:12:51.800849 ip-10-0-142-248 kubenswrapper[2561]: W0911 17:12:51.800820    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74c43b2d_c090_4a00_bc31_0c8783bc0633.slice/crio-53cfecef9e0522cd3b67557c86a3eba72b00cd3955bf008953ddaf72fbbe95ec WatchSource:0}: Error finding container 53cfecef9e0522cd3b67557c86a3eba72b00cd3955bf008953ddaf72fbbe95ec: Status 404 returned error can't find the container with id 53cfecef9e0522cd3b67557c86a3eba72b00cd3955bf008953ddaf72fbbe95ec
Sep 11 17:12:52.687969 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:52.687934    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2zgmz/must-gather-dmjxt" event={"ID":"74c43b2d-c090-4a00-bc31-0c8783bc0633","Type":"ContainerStarted","Data":"53cfecef9e0522cd3b67557c86a3eba72b00cd3955bf008953ddaf72fbbe95ec"}
Sep 11 17:12:58.727835 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:58.727791    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2zgmz/must-gather-dmjxt" event={"ID":"74c43b2d-c090-4a00-bc31-0c8783bc0633","Type":"ContainerStarted","Data":"7d264e3654c306cd20b85154507b26c2a41d557cd7ddfebd4404ca4c2d567ad2"}
Sep 11 17:12:58.728313 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:58.727847    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2zgmz/must-gather-dmjxt" event={"ID":"74c43b2d-c090-4a00-bc31-0c8783bc0633","Type":"ContainerStarted","Data":"5f30d892954d863bf35f36452faa6328641fd64b08cf614379c28d15dc63021a"}
Sep 11 17:12:58.744968 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:12:58.744912    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2zgmz/must-gather-dmjxt" podStartSLOduration=1.297519026 podStartE2EDuration="7.744891353s" totalImagesPullingTime="6.447372327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:12:58.742943646 +0000 UTC m=+3384.083037694" watchObservedRunningTime="2026-09-11 17:12:58.744891353 +0000 UTC m=+3384.084985399"
Sep 11 17:13:26.872764 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:26.872725    2561 generic.go:356] "Generic (PLEG): container finished" podID="74c43b2d-c090-4a00-bc31-0c8783bc0633" containerID="5f30d892954d863bf35f36452faa6328641fd64b08cf614379c28d15dc63021a" exitCode=0
Sep 11 17:13:26.873206 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:26.872808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2zgmz/must-gather-dmjxt" event={"ID":"74c43b2d-c090-4a00-bc31-0c8783bc0633","Type":"ContainerDied","Data":"5f30d892954d863bf35f36452faa6328641fd64b08cf614379c28d15dc63021a"}
Sep 11 17:13:26.873206 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:26.873164    2561 scope.go:117] "RemoveContainer" containerID="5f30d892954d863bf35f36452faa6328641fd64b08cf614379c28d15dc63021a"
Sep 11 17:13:27.374040 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:27.373953    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2zgmz_must-gather-dmjxt_74c43b2d-c090-4a00-bc31-0c8783bc0633/gather/0.log"
Sep 11 17:13:31.297492 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:31.297457    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xlpvp_b27b4250-c0d6-42d8-a2e5-4fc5aa796e2f/global-pull-secret-syncer/0.log"
Sep 11 17:13:31.421815 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:31.421787    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xwsg5_e86cc7cc-d70a-428d-abc9-4e68716d13d0/konnectivity-agent/0.log"
Sep 11 17:13:31.473149 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:31.473122    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-248.ec2.internal_e479a5e098468c317c85809b2c11ad9e/haproxy/0.log"
Sep 11 17:13:32.926054 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:32.926016    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-2zgmz/must-gather-dmjxt"]
Sep 11 17:13:32.926613 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:32.926230    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-2zgmz/must-gather-dmjxt" podUID="74c43b2d-c090-4a00-bc31-0c8783bc0633" containerName="copy" containerID="cri-o://7d264e3654c306cd20b85154507b26c2a41d557cd7ddfebd4404ca4c2d567ad2" gracePeriod=2
Sep 11 17:13:32.933117 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:32.933087    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-2zgmz/must-gather-dmjxt"]
Sep 11 17:13:33.172019 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.171998    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2zgmz_must-gather-dmjxt_74c43b2d-c090-4a00-bc31-0c8783bc0633/copy/0.log"
Sep 11 17:13:33.174698 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.174673    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2zgmz/must-gather-dmjxt"
Sep 11 17:13:33.176739 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:13:33.176691    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-dmjxt\" is forbidden: User \"system:node:ip-10-0-142-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2zgmz\": no relationship found between node 'ip-10-0-142-248.ec2.internal' and this object" podUID="74c43b2d-c090-4a00-bc31-0c8783bc0633" pod="openshift-must-gather-2zgmz/must-gather-dmjxt"
Sep 11 17:13:33.237729 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.237700    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf2qc\" (UniqueName: \"kubernetes.io/projected/74c43b2d-c090-4a00-bc31-0c8783bc0633-kube-api-access-lf2qc\") pod \"74c43b2d-c090-4a00-bc31-0c8783bc0633\" (UID: \"74c43b2d-c090-4a00-bc31-0c8783bc0633\") "
Sep 11 17:13:33.237844 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.237743    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/74c43b2d-c090-4a00-bc31-0c8783bc0633-must-gather-output\") pod \"74c43b2d-c090-4a00-bc31-0c8783bc0633\" (UID: \"74c43b2d-c090-4a00-bc31-0c8783bc0633\") "
Sep 11 17:13:33.239437 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.239394    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c43b2d-c090-4a00-bc31-0c8783bc0633-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "74c43b2d-c090-4a00-bc31-0c8783bc0633" (UID: "74c43b2d-c090-4a00-bc31-0c8783bc0633"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:13:33.240054 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.240031    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c43b2d-c090-4a00-bc31-0c8783bc0633-kube-api-access-lf2qc" (OuterVolumeSpecName: "kube-api-access-lf2qc") pod "74c43b2d-c090-4a00-bc31-0c8783bc0633" (UID: "74c43b2d-c090-4a00-bc31-0c8783bc0633"). InnerVolumeSpecName "kube-api-access-lf2qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:13:33.320930 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.320896    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74c43b2d-c090-4a00-bc31-0c8783bc0633" path="/var/lib/kubelet/pods/74c43b2d-c090-4a00-bc31-0c8783bc0633/volumes"
Sep 11 17:13:33.339296 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.339262    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf2qc\" (UniqueName: \"kubernetes.io/projected/74c43b2d-c090-4a00-bc31-0c8783bc0633-kube-api-access-lf2qc\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:13:33.339296 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.339292    2561 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/74c43b2d-c090-4a00-bc31-0c8783bc0633-must-gather-output\") on node \"ip-10-0-142-248.ec2.internal\" DevicePath \"\""
Sep 11 17:13:33.907638 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.907607    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2zgmz_must-gather-dmjxt_74c43b2d-c090-4a00-bc31-0c8783bc0633/copy/0.log"
Sep 11 17:13:33.907998 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.907973    2561 generic.go:356] "Generic (PLEG): container finished" podID="74c43b2d-c090-4a00-bc31-0c8783bc0633" containerID="7d264e3654c306cd20b85154507b26c2a41d557cd7ddfebd4404ca4c2d567ad2" exitCode=143
Sep 11 17:13:33.908073 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.908026    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2zgmz/must-gather-dmjxt"
Sep 11 17:13:33.908073 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.908056    2561 scope.go:117] "RemoveContainer" containerID="7d264e3654c306cd20b85154507b26c2a41d557cd7ddfebd4404ca4c2d567ad2"
Sep 11 17:13:33.916095 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.916073    2561 scope.go:117] "RemoveContainer" containerID="5f30d892954d863bf35f36452faa6328641fd64b08cf614379c28d15dc63021a"
Sep 11 17:13:33.930129 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.929958    2561 scope.go:117] "RemoveContainer" containerID="7d264e3654c306cd20b85154507b26c2a41d557cd7ddfebd4404ca4c2d567ad2"
Sep 11 17:13:33.930406 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:13:33.930274    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d264e3654c306cd20b85154507b26c2a41d557cd7ddfebd4404ca4c2d567ad2\": container with ID starting with 7d264e3654c306cd20b85154507b26c2a41d557cd7ddfebd4404ca4c2d567ad2 not found: ID does not exist" containerID="7d264e3654c306cd20b85154507b26c2a41d557cd7ddfebd4404ca4c2d567ad2"
Sep 11 17:13:33.930406 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.930303    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d264e3654c306cd20b85154507b26c2a41d557cd7ddfebd4404ca4c2d567ad2"} err="failed to get container status \"7d264e3654c306cd20b85154507b26c2a41d557cd7ddfebd4404ca4c2d567ad2\": rpc error: code = NotFound desc = could not find container \"7d264e3654c306cd20b85154507b26c2a41d557cd7ddfebd4404ca4c2d567ad2\": container with ID starting with 7d264e3654c306cd20b85154507b26c2a41d557cd7ddfebd4404ca4c2d567ad2 not found: ID does not exist"
Sep 11 17:13:33.930406 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.930324    2561 scope.go:117] "RemoveContainer" containerID="5f30d892954d863bf35f36452faa6328641fd64b08cf614379c28d15dc63021a"
Sep 11 17:13:33.930587 ip-10-0-142-248 kubenswrapper[2561]: E0911 17:13:33.930568    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f30d892954d863bf35f36452faa6328641fd64b08cf614379c28d15dc63021a\": container with ID starting with 5f30d892954d863bf35f36452faa6328641fd64b08cf614379c28d15dc63021a not found: ID does not exist" containerID="5f30d892954d863bf35f36452faa6328641fd64b08cf614379c28d15dc63021a"
Sep 11 17:13:33.930634 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:33.930592    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f30d892954d863bf35f36452faa6328641fd64b08cf614379c28d15dc63021a"} err="failed to get container status \"5f30d892954d863bf35f36452faa6328641fd64b08cf614379c28d15dc63021a\": rpc error: code = NotFound desc = could not find container \"5f30d892954d863bf35f36452faa6328641fd64b08cf614379c28d15dc63021a\": container with ID starting with 5f30d892954d863bf35f36452faa6328641fd64b08cf614379c28d15dc63021a not found: ID does not exist"
Sep 11 17:13:35.867772 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:35.867692    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5eed059-2d45-48e6-a379-3941236f35fd/alertmanager/0.log"
Sep 11 17:13:35.903895 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:35.903870    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5eed059-2d45-48e6-a379-3941236f35fd/config-reloader/0.log"
Sep 11 17:13:35.938291 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:35.938267    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5eed059-2d45-48e6-a379-3941236f35fd/kube-rbac-proxy-web/0.log"
Sep 11 17:13:35.976342 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:35.976312    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5eed059-2d45-48e6-a379-3941236f35fd/kube-rbac-proxy/0.log"
Sep 11 17:13:36.009516 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.009475    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5eed059-2d45-48e6-a379-3941236f35fd/kube-rbac-proxy-metric/0.log"
Sep 11 17:13:36.043318 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.043288    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5eed059-2d45-48e6-a379-3941236f35fd/prom-label-proxy/0.log"
Sep 11 17:13:36.077893 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.077864    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5eed059-2d45-48e6-a379-3941236f35fd/init-config-reloader/0.log"
Sep 11 17:13:36.131834 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.131759    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-2dmxt_ba15f5ca-5cac-48e7-9622-e3166bdc1dca/cluster-monitoring-operator/0.log"
Sep 11 17:13:36.162473 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.162439    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-88cgp_87a63b67-349c-46c4-8123-c2413ceafd6f/kube-state-metrics/0.log"
Sep 11 17:13:36.189778 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.189753    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-88cgp_87a63b67-349c-46c4-8123-c2413ceafd6f/kube-rbac-proxy-main/0.log"
Sep 11 17:13:36.221532 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.221503    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-88cgp_87a63b67-349c-46c4-8123-c2413ceafd6f/kube-rbac-proxy-self/0.log"
Sep 11 17:13:36.293496 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.293465    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-49rm9_b08d9859-821d-4f5f-8058-eaf19c990774/monitoring-plugin/0.log"
Sep 11 17:13:36.517922 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.517897    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xlhbf_6084648b-e681-43b3-b0ea-28a6efef0b7c/node-exporter/0.log"
Sep 11 17:13:36.548120 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.548085    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xlhbf_6084648b-e681-43b3-b0ea-28a6efef0b7c/kube-rbac-proxy/0.log"
Sep 11 17:13:36.584945 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.584922    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xlhbf_6084648b-e681-43b3-b0ea-28a6efef0b7c/init-textfile/0.log"
Sep 11 17:13:36.621640 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.621615    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-xhkkp_ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2/kube-rbac-proxy-main/0.log"
Sep 11 17:13:36.649052 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.649028    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-xhkkp_ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2/kube-rbac-proxy-self/0.log"
Sep 11 17:13:36.676159 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.676135    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-xhkkp_ba013ca8-ec3b-4ce8-940d-620cb9c7d3a2/openshift-state-metrics/0.log"
Sep 11 17:13:36.719697 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.719672    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_786b414e-7f35-4733-a87c-f0b7c32928b9/prometheus/0.log"
Sep 11 17:13:36.743692 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.743671    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_786b414e-7f35-4733-a87c-f0b7c32928b9/config-reloader/0.log"
Sep 11 17:13:36.779326 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.779261    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_786b414e-7f35-4733-a87c-f0b7c32928b9/thanos-sidecar/0.log"
Sep 11 17:13:36.811226 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.811200    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_786b414e-7f35-4733-a87c-f0b7c32928b9/kube-rbac-proxy-web/0.log"
Sep 11 17:13:36.840867 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.840839    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_786b414e-7f35-4733-a87c-f0b7c32928b9/kube-rbac-proxy/0.log"
Sep 11 17:13:36.871076 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.871051    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_786b414e-7f35-4733-a87c-f0b7c32928b9/kube-rbac-proxy-thanos/0.log"
Sep 11 17:13:36.906828 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.906802    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_786b414e-7f35-4733-a87c-f0b7c32928b9/init-config-reloader/0.log"
Sep 11 17:13:36.951441 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.951411    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-tnbqj_3626db7d-bc08-4ba1-b3e8-88eff89e0485/prometheus-operator/0.log"
Sep 11 17:13:36.977118 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:36.977091    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-tnbqj_3626db7d-bc08-4ba1-b3e8-88eff89e0485/kube-rbac-proxy/0.log"
Sep 11 17:13:37.022338 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:37.022303    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-h46t5_b13128d0-b6b9-4b72-ae5f-8d638cd5b97c/prometheus-operator-admission-webhook/0.log"
Sep 11 17:13:37.086962 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:37.086879    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68d6c68898-tstt8_506cf99c-d273-4156-b63a-e28179d3270d/telemeter-client/0.log"
Sep 11 17:13:37.143909 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:37.143878    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68d6c68898-tstt8_506cf99c-d273-4156-b63a-e28179d3270d/reload/0.log"
Sep 11 17:13:37.178705 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:37.178681    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68d6c68898-tstt8_506cf99c-d273-4156-b63a-e28179d3270d/kube-rbac-proxy/0.log"
Sep 11 17:13:37.237553 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:37.237527    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c75b4f8f-xqr2k_b15f23b5-8a1e-4763-838c-ba6550c4918d/thanos-query/0.log"
Sep 11 17:13:37.274429 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:37.274397    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c75b4f8f-xqr2k_b15f23b5-8a1e-4763-838c-ba6550c4918d/kube-rbac-proxy-web/0.log"
Sep 11 17:13:37.306369 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:37.306342    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c75b4f8f-xqr2k_b15f23b5-8a1e-4763-838c-ba6550c4918d/kube-rbac-proxy/0.log"
Sep 11 17:13:37.336138 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:37.336113    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c75b4f8f-xqr2k_b15f23b5-8a1e-4763-838c-ba6550c4918d/prom-label-proxy/0.log"
Sep 11 17:13:37.367458 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:37.367383    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c75b4f8f-xqr2k_b15f23b5-8a1e-4763-838c-ba6550c4918d/kube-rbac-proxy-rules/0.log"
Sep 11 17:13:37.401873 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:37.401847    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c75b4f8f-xqr2k_b15f23b5-8a1e-4763-838c-ba6550c4918d/kube-rbac-proxy-metrics/0.log"
Sep 11 17:13:38.558921 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:38.558847    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-kgmlh_06ff2972-636e-4ca7-afcb-2c8e1cc76e0d/networking-console-plugin/0.log"
Sep 11 17:13:39.147030 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.146997    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"]
Sep 11 17:13:39.147394 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.147380    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c43b2d-c090-4a00-bc31-0c8783bc0633" containerName="gather"
Sep 11 17:13:39.147443 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.147395    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c43b2d-c090-4a00-bc31-0c8783bc0633" containerName="gather"
Sep 11 17:13:39.147443 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.147409    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c43b2d-c090-4a00-bc31-0c8783bc0633" containerName="copy"
Sep 11 17:13:39.147443 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.147415    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c43b2d-c090-4a00-bc31-0c8783bc0633" containerName="copy"
Sep 11 17:13:39.147552 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.147471    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c43b2d-c090-4a00-bc31-0c8783bc0633" containerName="copy"
Sep 11 17:13:39.147552 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.147482    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c43b2d-c090-4a00-bc31-0c8783bc0633" containerName="gather"
Sep 11 17:13:39.152580 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.152558    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.154760 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.154732    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w5hdz\"/\"kube-root-ca.crt\""
Sep 11 17:13:39.154871 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.154734    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w5hdz\"/\"openshift-service-ca.crt\""
Sep 11 17:13:39.155134 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.155119    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w5hdz\"/\"default-dockercfg-4gl5p\""
Sep 11 17:13:39.160299 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.160276    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"]
Sep 11 17:13:39.188836 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.188803    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10b596dc-b93d-48d6-9ddc-0c0f2e6179f9-lib-modules\") pod \"perf-node-gather-daemonset-8dkz6\" (UID: \"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.189043 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.188853    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/10b596dc-b93d-48d6-9ddc-0c0f2e6179f9-proc\") pod \"perf-node-gather-daemonset-8dkz6\" (UID: \"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.189043 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.188957    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/10b596dc-b93d-48d6-9ddc-0c0f2e6179f9-podres\") pod \"perf-node-gather-daemonset-8dkz6\" (UID: \"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.189043 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.188986    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhbvd\" (UniqueName: \"kubernetes.io/projected/10b596dc-b93d-48d6-9ddc-0c0f2e6179f9-kube-api-access-zhbvd\") pod \"perf-node-gather-daemonset-8dkz6\" (UID: \"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.189208 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.189066    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10b596dc-b93d-48d6-9ddc-0c0f2e6179f9-sys\") pod \"perf-node-gather-daemonset-8dkz6\" (UID: \"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.289801 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.289759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10b596dc-b93d-48d6-9ddc-0c0f2e6179f9-lib-modules\") pod \"perf-node-gather-daemonset-8dkz6\" (UID: \"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.289801 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.289803    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/10b596dc-b93d-48d6-9ddc-0c0f2e6179f9-proc\") pod \"perf-node-gather-daemonset-8dkz6\" (UID: \"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.290034 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.289851    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/10b596dc-b93d-48d6-9ddc-0c0f2e6179f9-podres\") pod \"perf-node-gather-daemonset-8dkz6\" (UID: \"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.290034 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.289868    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhbvd\" (UniqueName: \"kubernetes.io/projected/10b596dc-b93d-48d6-9ddc-0c0f2e6179f9-kube-api-access-zhbvd\") pod \"perf-node-gather-daemonset-8dkz6\" (UID: \"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.290034 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.289898    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10b596dc-b93d-48d6-9ddc-0c0f2e6179f9-sys\") pod \"perf-node-gather-daemonset-8dkz6\" (UID: \"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.290034 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.289960    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/10b596dc-b93d-48d6-9ddc-0c0f2e6179f9-proc\") pod \"perf-node-gather-daemonset-8dkz6\" (UID: \"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.290034 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.289963    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10b596dc-b93d-48d6-9ddc-0c0f2e6179f9-lib-modules\") pod \"perf-node-gather-daemonset-8dkz6\" (UID: \"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.290034 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.289970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10b596dc-b93d-48d6-9ddc-0c0f2e6179f9-sys\") pod \"perf-node-gather-daemonset-8dkz6\" (UID: \"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.290034 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.290010    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/10b596dc-b93d-48d6-9ddc-0c0f2e6179f9-podres\") pod \"perf-node-gather-daemonset-8dkz6\" (UID: \"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.298806 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.298780    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhbvd\" (UniqueName: \"kubernetes.io/projected/10b596dc-b93d-48d6-9ddc-0c0f2e6179f9-kube-api-access-zhbvd\") pod \"perf-node-gather-daemonset-8dkz6\" (UID: \"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9\") " pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.463119 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.463076    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.734434 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.734403    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"]
Sep 11 17:13:39.737169 ip-10-0-142-248 kubenswrapper[2561]: W0911 17:13:39.737136    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod10b596dc_b93d_48d6_9ddc_0c0f2e6179f9.slice/crio-cd484adba232523cec337f0b39733d7f29e5b1eb92ee6813f2dcb8fbbe8f96cf WatchSource:0}: Error finding container cd484adba232523cec337f0b39733d7f29e5b1eb92ee6813f2dcb8fbbe8f96cf: Status 404 returned error can't find the container with id cd484adba232523cec337f0b39733d7f29e5b1eb92ee6813f2dcb8fbbe8f96cf
Sep 11 17:13:39.936669 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.936630    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6" event={"ID":"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9","Type":"ContainerStarted","Data":"f6379d146f2536372e780ad4f88ced89f601ade9a7c698b10d0744fe9735c420"}
Sep 11 17:13:39.936839 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.936685    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6" event={"ID":"10b596dc-b93d-48d6-9ddc-0c0f2e6179f9","Type":"ContainerStarted","Data":"cd484adba232523cec337f0b39733d7f29e5b1eb92ee6813f2dcb8fbbe8f96cf"}
Sep 11 17:13:39.936839 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.936710    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:39.953838 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:39.953788    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6" podStartSLOduration=0.953770558 podStartE2EDuration="953.770558ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:13:39.952414877 +0000 UTC m=+3425.292508924" watchObservedRunningTime="2026-09-11 17:13:39.953770558 +0000 UTC m=+3425.293864650"
Sep 11 17:13:41.043777 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:41.043747    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r7twx_cc5f30a6-4339-4955-baea-2c123c7a1db9/dns/0.log"
Sep 11 17:13:41.071375 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:41.071347    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r7twx_cc5f30a6-4339-4955-baea-2c123c7a1db9/kube-rbac-proxy/0.log"
Sep 11 17:13:41.200390 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:41.200360    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-txkjh_062d3c49-c0e6-400d-bf42-5ef08d0c83d7/dns-node-resolver/0.log"
Sep 11 17:13:41.719930 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:41.719900    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2lrtq_f76398de-0983-4cee-afab-0e48e5b8db38/node-ca/0.log"
Sep 11 17:13:42.927420 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:42.927387    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bnxmw_097bce6a-44cf-4a1d-af40-9681a6ced969/serve-healthcheck-canary/0.log"
Sep 11 17:13:43.454465 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:43.454441    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ttkdv_35938125-0fd6-46f4-80f4-04f615985a5b/kube-rbac-proxy/0.log"
Sep 11 17:13:43.478274 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:43.478247    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ttkdv_35938125-0fd6-46f4-80f4-04f615985a5b/exporter/0.log"
Sep 11 17:13:43.506148 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:43.506118    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ttkdv_35938125-0fd6-46f4-80f4-04f615985a5b/extractor/0.log"
Sep 11 17:13:45.951754 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:45.951720    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w5hdz/perf-node-gather-daemonset-8dkz6"
Sep 11 17:13:46.346096 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:46.346019    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-j8rk2_f1a90201-ed37-43bd-a91f-27460661fa37/s3-init/0.log"
Sep 11 17:13:46.365473 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:46.365449    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-j8rk2_f1a90201-ed37-43bd-a91f-27460661fa37/download-hf-model/0.log"
Sep 11 17:13:46.396126 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:46.396103    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-9ztlx_ce8738b9-903b-4009-9bf3-49de8955d907/s3-tls-init-custom/0.log"
Sep 11 17:13:46.415903 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:46.415879    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-9ztlx_ce8738b9-903b-4009-9bf3-49de8955d907/download-hf-model/0.log"
Sep 11 17:13:46.448023 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:46.447983    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-f2pml_a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b/s3-tls-init-serving/0.log"
Sep 11 17:13:46.467528 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:46.467494    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-f2pml_a0bf062c-70a6-4f68-8e4b-74aa7a5ce86b/download-hf-model/0.log"
Sep 11 17:13:52.530072 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:52.530039    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5jvnz_346d7ed3-37a8-43fd-8544-96f9bc2fb32f/kube-multus/0.log"
Sep 11 17:13:52.815739 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:52.815661    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xmqw_53bba805-9236-4e17-bcb2-a4b29b4a2355/kube-multus-additional-cni-plugins/0.log"
Sep 11 17:13:52.841582 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:52.841532    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xmqw_53bba805-9236-4e17-bcb2-a4b29b4a2355/egress-router-binary-copy/0.log"
Sep 11 17:13:52.871939 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:52.871906    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xmqw_53bba805-9236-4e17-bcb2-a4b29b4a2355/cni-plugins/0.log"
Sep 11 17:13:52.898467 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:52.898439    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xmqw_53bba805-9236-4e17-bcb2-a4b29b4a2355/bond-cni-plugin/0.log"
Sep 11 17:13:52.932120 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:52.932093    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xmqw_53bba805-9236-4e17-bcb2-a4b29b4a2355/routeoverride-cni/0.log"
Sep 11 17:13:52.956624 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:52.956597    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xmqw_53bba805-9236-4e17-bcb2-a4b29b4a2355/whereabouts-cni-bincopy/0.log"
Sep 11 17:13:52.982990 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:52.982967    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xmqw_53bba805-9236-4e17-bcb2-a4b29b4a2355/whereabouts-cni/0.log"
Sep 11 17:13:53.292693 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:53.292655    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-z6xtq_b15c0e16-1077-454c-aafa-16902113243c/network-metrics-daemon/0.log"
Sep 11 17:13:53.327106 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:53.327082    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-z6xtq_b15c0e16-1077-454c-aafa-16902113243c/kube-rbac-proxy/0.log"
Sep 11 17:13:54.661189 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:54.661144    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-controller/0.log"
Sep 11 17:13:54.683850 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:54.683811    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/0.log"
Sep 11 17:13:54.684043 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:54.684027    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovn-acl-logging/1.log"
Sep 11 17:13:54.712675 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:54.712652    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/kube-rbac-proxy-node/0.log"
Sep 11 17:13:54.742460 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:54.742438    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/kube-rbac-proxy-ovn-metrics/0.log"
Sep 11 17:13:54.766621 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:54.766594    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/northd/0.log"
Sep 11 17:13:54.791359 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:54.791316    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/nbdb/0.log"
Sep 11 17:13:54.824346 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:54.824322    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/sbdb/0.log"
Sep 11 17:13:54.876642 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:54.876615    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sz5sg_38d0a297-96bb-437e-ac8d-7ef56f3c3c87/ovnkube-controller/0.log"
Sep 11 17:13:56.014521 ip-10-0-142-248 kubenswrapper[2561]: I0911 17:13:56.014493    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qdmr8_af5da358-605f-4f1f-a20a-5aa63fb8889d/network-check-target-container/0.log"