Sep 17 20:47:29.788122 ip-10-0-131-94 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 20:47:29.788136 ip-10-0-131-94 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 20:47:29.788144 ip-10-0-131-94 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 20:47:29.788422 ip-10-0-131-94 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 20:47:39.871376 ip-10-0-131-94 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 20:47:39.871397 ip-10-0-131-94 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 8215f68c06b447fab2984fc531952651 --
Sep 17 20:49:47.050821 ip-10-0-131-94 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 20:49:50.437394 ip-10-0-131-94 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:50.437394 ip-10-0-131-94 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 20:49:50.437394 ip-10-0-131-94 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:50.437394 ip-10-0-131-94 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 20:49:50.437394 ip-10-0-131-94 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:50.438091 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.437469    2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 20:49:50.441233 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441218    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:50.441233 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441233    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441236    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441240    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441242    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441245    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441251    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441255    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441258    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441262    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441264    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441268    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441273    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441275    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441283    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441286    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441289    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441291    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441294    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441296    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:50.441295 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441299    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441302    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441305    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441308    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441310    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441313    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441315    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441318    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441321    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441323    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441325    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441328    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441330    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441333    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441335    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441338    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441340    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441343    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441345    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441348    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:50.441757 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441350    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441353    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441355    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441358    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441360    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441362    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441365    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441367    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441369    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441372    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441374    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441377    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441379    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441383    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441389    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441392    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441396    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441399    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441402    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:50.442232 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441404    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441407    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441409    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441412    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441414    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441417    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441419    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441422    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441425    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441427    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441429    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441432    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441435    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441438    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441441    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441443    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441446    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441448    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441451    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:50.442709 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441453    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441456    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441458    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441462    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441465    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441467    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441470    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441472    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441475    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441478    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441481    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441483    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441486    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441488    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441491    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441493    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441495    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441498    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441500    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441503    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:50.443164 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.441505    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442247    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442255    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442260    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442264    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442267    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442271    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442274    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442276    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442279    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442281    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442284    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442286    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442289    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442291    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442294    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442297    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442300    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442302    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442305    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:50.443703 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442308    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442311    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442314    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442316    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442319    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442322    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442324    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442327    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442329    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442331    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442334    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442337    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442339    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442342    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442345    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442349    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442352    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442354    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442357    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442361    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:50.444196 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442363    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442368    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442371    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442374    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442376    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442379    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442381    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442384    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442386    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442389    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442391    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442394    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442397    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442399    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442402    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442404    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442407    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442409    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442412    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:50.444694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442414    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442417    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442419    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442421    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442424    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442427    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442429    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442431    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442434    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442436    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442438    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442441    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442443    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442446    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442449    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442451    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442454    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442457    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442460    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442462    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:50.445152 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442467    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442470    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442472    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442475    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442477    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442480    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442483    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442485    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442488    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442490    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442492    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442495    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442498    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442500    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442503    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442505    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442507    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442510    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442513    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442515    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:50.445684 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442517    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442907    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442912    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442915    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442918    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442921    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442924    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442927    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442930    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442932    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442935    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442937    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442942    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442944    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442947    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442949    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442952    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442954    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442957    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442960    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:50.446190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442962    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442965    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442968    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442970    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442973    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442975    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442979    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442983    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442985    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442988    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442990    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442993    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442995    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.442998    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443000    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443003    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443005    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443008    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443014    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443016    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:50.446699 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443019    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443021    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443024    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443026    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443029    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443032    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443034    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443037    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443039    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443044    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443046    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443049    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443051    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443054    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443056    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443059    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443061    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443063    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443066    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443068    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:50.447195 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443071    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443073    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443076    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443078    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443081    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443083    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443085    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443088    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443090    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443093    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443096    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443099    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443101    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443104    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443106    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443109    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443113    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443115    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443117    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443120    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:50.447694 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443123    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443126    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443129    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443132    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443135    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443137    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443142    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443146    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443149    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443153    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443156    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443159    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443162    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443164    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443167    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443170    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443173    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443176    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443179    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:50.448187 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443182    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443256    2571 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443287    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443295    2571 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443300    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443305    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443308    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443313    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443317    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443321    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443324    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443327    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443330    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443333    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443337    2571 flags.go:64] FLAG: --cgroup-root=""
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443339    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443342    2571 flags.go:64] FLAG: --client-ca-file=""
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443345    2571 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443348    2571 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443353    2571 flags.go:64] FLAG: --cluster-domain=""
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443355    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443358    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443361    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443364    2571 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 20:49:50.448694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443368    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443371    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443374    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443377    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443380    2571 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443382    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443385    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443388    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443390    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443394    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443397    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443400    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443402    2571 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443405    2571 flags.go:64] FLAG: --enable-server="true"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443409    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443414    2571 flags.go:64] FLAG: --event-burst="100"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443417    2571 flags.go:64] FLAG: --event-qps="50"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443420    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443423    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443426    2571 flags.go:64] FLAG: --eviction-hard=""
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443429    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443432    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443435    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443438    2571 flags.go:64] FLAG: --eviction-soft=""
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443441    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 20:49:50.449258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443444    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443447    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443450    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443453    2571 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443455    2571 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443458    2571 flags.go:64] FLAG: --feature-gates=""
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443462    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443465    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443468    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443471    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443473    2571 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443476    2571 flags.go:64] FLAG: --help="false"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443479    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-131-94.ec2.internal"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443482    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443484    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443487    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443490    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443493    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443496    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443499    2571 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443502    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443505    2571 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443508    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443511    2571 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 20:49:50.449947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443513    2571 flags.go:64] FLAG: --kube-reserved=""
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443516    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443519    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443522    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443524    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443527    2571 flags.go:64] FLAG: --lock-file=""
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443530    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443533    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443535    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443541    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443543    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443560    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443563    2571 flags.go:64] FLAG: --logging-format="text"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443566    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443569    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443572    2571 flags.go:64] FLAG: --manifest-url=""
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443575    2571 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443579    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443582    2571 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443586    2571 flags.go:64] FLAG: --max-pods="110"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443589    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443591    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443595    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443598    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443600    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 20:49:50.450911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443603    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443606    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443614    2571 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443617    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443620    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443623    2571 flags.go:64] FLAG: --pod-cidr=""
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443626    2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443632    2571 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443635    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443638    2571 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443640    2571 flags.go:64] FLAG: --port="10250"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443643    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443646    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bdfa26ff7248ae33"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443649    2571 flags.go:64] FLAG: --qos-reserved=""
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443652    2571 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443655    2571 flags.go:64] FLAG: --register-node="true"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443658    2571 flags.go:64] FLAG: --register-with-taints=""
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443661    2571 flags.go:64] FLAG: --registry-burst="10"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443664    2571 flags.go:64] FLAG: --registry-qps="5"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443666    2571 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443669    2571 flags.go:64] FLAG: --reserved-memory=""
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443672    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443675    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443678    2571 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443681    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 20:49:50.451993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443683    2571 flags.go:64] FLAG: --runonce="false"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443686    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443689    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443691    2571 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443694    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443697    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443700    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443703    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443706    2571 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443709    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443715    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443718    2571 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443721    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443724    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443727    2571 flags.go:64] FLAG: --system-cgroups=""
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443730    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443736    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443739    2571 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443741    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443747    2571 flags.go:64] FLAG: --tls-min-version=""
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443750    2571 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443753    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443756    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443758    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443761    2571 flags.go:64] FLAG: --v="2"
Sep 17 20:49:50.453079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443765    2571 flags.go:64] FLAG: --version="false"
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443769    2571 flags.go:64] FLAG: --vmodule=""
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443772    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.443775    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443886    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443890    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443893    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443896    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443899    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443901    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443904    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443907    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443909    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443912    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443914    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443917    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443919    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443923    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443926    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443929    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:50.454032 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443932    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443934    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443937    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443940    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443942    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443946    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443951    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443955    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443959    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443962    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443965    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443968    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443970    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443973    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443975    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443978    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443981    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443983    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443986    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443989    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:50.454766 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443991    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443996    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.443999    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444002    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444005    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444008    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444010    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444013    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444015    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444019    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444022    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444025    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444028    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444031    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444034    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444037    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444040    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444042    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444046    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:50.455619 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444048    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444051    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444053    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444056    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444058    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444060    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444063    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444065    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444068    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444070    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444073    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444075    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444078    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444080    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444083    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444085    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444088    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444090    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444092    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444095    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:50.456185 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444097    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444101    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444105    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444108    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444111    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444114    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444117    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444120    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444122    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444125    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444127    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444132    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444135    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444138    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444141    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444144    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444146    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444149    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444152    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:50.456691 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444155    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:50.457212 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444158    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:50.457212 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444160    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:50.457212 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444163    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:50.457212 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.444166    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:50.457212 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.444865    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:50.457212 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.452332    2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 20:49:50.457212 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.452351    2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 20:49:50.457212 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.452384    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 20:49:50.457212 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.452397    2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 20:49:50.457212 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452482    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:50.457212 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452489    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:50.457212 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452494    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:50.457212 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452498    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:50.457212 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452503    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:50.457212 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452507    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452511    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452515    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452520    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452524    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452528    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452532    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452536    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452541    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452564    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452569    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452573    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452577    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452581    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452585    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452588    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452593    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452597    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452601    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452605    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:50.457804 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452609    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452613    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452617    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452621    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452628    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452633    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452637    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452642    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452646    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452651    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452655    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452659    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452663    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452667    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452671    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452675    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452679    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452686    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:50.458329 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452692    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452698    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452704    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452711    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452715    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452720    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452725    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452729    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452733    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452737    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452741    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452745    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452748    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452753    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452757    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452761    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452765    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452769    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452774    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:50.458784 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452779    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452784    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452788    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452792    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452796    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452800    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452805    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452808    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452813    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452817    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452821    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452825    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452829    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452833    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452838    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452841    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452845    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452849    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452853    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452857    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:50.459302 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452861    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:50.460062 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452865    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:50.460062 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452869    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:50.460062 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452873    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:50.460062 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452877    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:50.460062 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452881    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:50.460062 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452885    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:50.460062 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452889    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:50.460062 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452893    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:50.460062 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452897    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:50.460062 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452901    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:50.460062 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452906    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:50.460062 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452911    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:50.460062 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452920    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:50.460062 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452924    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:50.460062 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452929    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:50.460062 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.452933    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:50.460516 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.452940    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:50.460516 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453114    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:50.460516 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453122    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:50.460516 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453127    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:50.460516 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453131    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:50.460516 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453135    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:50.460516 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453140    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:50.460516 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453144    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:50.460516 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453148    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:50.460516 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453152    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:50.460516 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453156    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:50.460516 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453160    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:50.460516 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453164    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:50.460516 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453168    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:50.460516 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453172    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453176    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453180    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453184    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453188    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453192    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453197    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453201    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453205    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453209    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453214    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453218    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453222    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453226    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453230    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453235    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453240    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453244    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453248    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453252    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:50.460983 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453258    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453264    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453268    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453272    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453276    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453280    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453284    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453288    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453293    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453297    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453303    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453309    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453314    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453318    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453322    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453326    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453331    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453335    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453339    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453343    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:50.461867 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453347    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453351    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453355    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453359    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453365    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453371    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453376    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453381    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453386    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453390    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453394    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453399    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453403    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453406    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453411    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453414    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453418    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453422    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453426    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:50.462530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453430    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453434    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453438    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453442    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453446    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453449    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453453    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453457    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453462    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453466    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453470    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453474    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453477    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453484    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453488    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453492    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453497    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453501    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453505    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453509    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:50.463294 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453513    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:50.463933 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453517    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:50.463933 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453522    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:50.463933 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453526    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:50.463933 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453530    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:50.463933 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453533    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:50.463933 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:50.453537    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:50.463933 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.453565    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:50.463933 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.453718    2571 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 20:49:50.463933 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.457401    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 20:49:50.463933 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.458945    2571 server.go:1019] "Starting client certificate rotation"
Sep 17 20:49:50.463933 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.459046    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 20:49:50.463933 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.459086    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 20:49:50.463933 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.461422    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 20:49:50.467888 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.467862    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 20:49:50.471900 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.471877    2571 log.go:25] "Validated CRI v1 runtime API"
Sep 17 20:49:50.473951 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.473934    2571 log.go:25] "Validated CRI v1 image API"
Sep 17 20:49:50.476069 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.476054    2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 20:49:50.478122 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.478098    2571 fs.go:135] Filesystem UUIDs: map[298a4009-5458-4296-b403-f5ac78c23f8e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bc9837e8-8d41-48f1-b684-9648895893e8:/dev/nvme0n1p3]
Sep 17 20:49:50.478201 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.478121    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 20:49:50.482372 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.482260    2571 manager.go:216] Machine: {Timestamp:2026-09-17 20:49:50.481795871 +0000 UTC m=+1.986273458 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101478 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22ffce0c4562d18409001c09261b7c SystemUUID:ec22ffce-0c45-62d1-8409-001c09261b7c BootID:8215f68c-06b4-47fa-b298-4fc531952651 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e0:24:3a:8f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e0:24:3a:8f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:f4:ee:60:ab:93 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 20:49:50.482372 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.482363    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 20:49:50.482501 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.482435    2571 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 20:49:50.484411 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.484383    2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 20:49:50.484572 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.484415    2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-94.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 20:49:50.485492 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.485482    2571 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 20:49:50.485523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.485495    2571 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 20:49:50.485523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.485503    2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 20:49:50.485523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.485513    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 20:49:50.485622 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.485601    2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 20:49:50.488311 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.488292    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 20:49:50.490013 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.490001    2571 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 20:49:50.490155 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.490145    2571 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 20:49:50.490194 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.490176    2571 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 20:49:50.490194 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.490192    2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 20:49:50.490254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.490207    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 20:49:50.490254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.490222    2571 kubelet.go:398] "Adding apiserver pod source"
Sep 17 20:49:50.490254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.490234    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 20:49:50.490629 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.490617    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 20:49:50.490674 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.490636    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 20:49:50.492960 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.492939    2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 20:49:50.494878 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.494861    2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 20:49:50.494943 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.494895    2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 20:49:50.495763 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.495751    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 20:49:50.495800 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.495769    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 20:49:50.495800 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.495775    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 20:49:50.495854 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.495805    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 20:49:50.495854 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.495819    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 20:49:50.495854 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.495826    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 20:49:50.495854 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.495840    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 20:49:50.495854 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.495845    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 20:49:50.495854 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.495852    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 20:49:50.495854 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.495857    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 20:49:50.496030 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.495872    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 20:49:50.496030 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.495878    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 20:49:50.496030 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.495913    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 20:49:50.496030 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.495919    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 20:49:50.499506 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.499492    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-94.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 20:49:50.502590 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:50.500493    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 20:49:50.502690 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.502599    2571 server.go:1268] "Started kubelet"
Sep 17 20:49:50.502690 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:50.500491    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 20:49:50.503153 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.503109    2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 20:49:50.503424 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.503255    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 20:49:50.503424 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.503323    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 20:49:50.503711 ip-10-0-131-94 systemd[1]: Started Kubernetes Kubelet.
Sep 17 20:49:50.505072 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.505007    2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 20:49:50.505693 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.505676    2571 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 20:49:50.534294 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.534269    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-76vln"
Sep 17 20:49:50.534294 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.534295    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-76vln"
Sep 17 20:49:50.534870 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:50.534848    2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 20:49:50.536830 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:50.535969    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-94.ec2.internal.18d636d5072aba31  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-94.ec2.internal,UID:ip-10-0-131-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-94.ec2.internal,},FirstTimestamp:2026-09-17 20:49:50.502525489 +0000 UTC m=+2.007003081,LastTimestamp:2026-09-17 20:49:50.502525489 +0000 UTC m=+2.007003081,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-94.ec2.internal,}"
Sep 17 20:49:50.539241 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.539225    2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 20:49:50.539328 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.539244    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 20:49:50.539328 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:50.539312    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-94.ec2.internal\" not found"
Sep 17 20:49:50.539754 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.539733    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 20:49:50.539999 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.539980    2571 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 20:49:50.540087 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.540003    2571 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 20:49:50.540200 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.540146    2571 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 20:49:50.540200 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.540158    2571 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 20:49:50.540364 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.540347    2571 factory.go:55] Registering systemd factory
Sep 17 20:49:50.540431 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.540367    2571 factory.go:223] Registration of the systemd container factory successfully
Sep 17 20:49:50.540592 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.540576    2571 factory.go:153] Registering CRI-O factory
Sep 17 20:49:50.540592 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.540590    2571 factory.go:223] Registration of the crio container factory successfully
Sep 17 20:49:50.540724 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.540638    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 20:49:50.540724 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.540657    2571 factory.go:103] Registering Raw factory
Sep 17 20:49:50.540724 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.540684    2571 manager.go:1196] Started watching for new ooms in manager
Sep 17 20:49:50.541160 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.541146    2571 manager.go:349] Starting recovery of all containers
Sep 17 20:49:50.550030 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.550009    2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:50.550030 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.550023    2571 manager.go:354] Recovery completed
Sep 17 20:49:50.552677 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:50.552654    2571 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-94.ec2.internal\" not found" node="ip-10-0-131-94.ec2.internal"
Sep 17 20:49:50.554106 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.554093    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:50.556040 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.556026    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:50.556120 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.556061    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:50.556120 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.556075    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:50.556721 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.556709    2571 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 20:49:50.556780 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.556721    2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 20:49:50.556780 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.556741    2571 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 20:49:50.556780 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.556750    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:50.557482 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.557470    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:50.557537 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.557500    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:50.557537 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.557514    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:50.558802 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.558789    2571 policy_none.go:49] "None policy: Start"
Sep 17 20:49:50.558842 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.558805    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 20:49:50.558842 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.558816    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 20:49:50.560656 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.560644    2571 policy_none.go:47] "Start"
Sep 17 20:49:50.611360 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.602750    2571 manager.go:337] "Starting Device Plugin manager"
Sep 17 20:49:50.611360 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:50.602810    2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 20:49:50.611360 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.602821    2571 server.go:87] "Starting device plugin registration server"
Sep 17 20:49:50.611360 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.603068    2571 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 20:49:50.611360 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.603087    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 20:49:50.611360 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.603294    2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 20:49:50.611360 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.603945    2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 20:49:50.611360 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.603957    2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 20:49:50.611360 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:50.606511    2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 20:49:50.611360 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:50.606571    2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-94.ec2.internal\" not found"
Sep 17 20:49:50.662132 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.662092    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 20:49:50.663443 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.663421    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 20:49:50.663537 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.663464    2571 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 20:49:50.663537 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.663490    2571 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 20:49:50.663537 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:50.663524    2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 20:49:50.666724 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.666703    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:50.703695 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.703650    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:50.704526 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.704511    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:50.704619 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.704541    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:50.704619 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.704566    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:50.704619 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.704589    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-94.ec2.internal"
Sep 17 20:49:50.710720 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.710702    2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-94.ec2.internal"
Sep 17 20:49:50.710811 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:50.710731    2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-94.ec2.internal\": node \"ip-10-0-131-94.ec2.internal\" not found"
Sep 17 20:49:50.720640 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:50.720623    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-94.ec2.internal\" not found"
Sep 17 20:49:50.764138 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.764092    2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-94.ec2.internal"]
Sep 17 20:49:50.764260 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.764182    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:50.765936 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.765911    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:50.766044 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.765953    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:50.766044 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.765966    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:50.768139 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.768125    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:50.768282 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.768256    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal"
Sep 17 20:49:50.768331 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.768308    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:50.769071 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.769056    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:50.769154 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.769081    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:50.769154 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.769087    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:50.769154 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.769115    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:50.769154 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.769130    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:50.769154 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.769090    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:50.771407 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.771388    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.ec2.internal"
Sep 17 20:49:50.771491 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.771420    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:50.772244 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.772225    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:50.772331 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.772254    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:50.772331 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.772270    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:50.802235 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:50.802208    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-94.ec2.internal\" not found" node="ip-10-0-131-94.ec2.internal"
Sep 17 20:49:50.806674 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:50.806658    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-94.ec2.internal\" not found" node="ip-10-0-131-94.ec2.internal"
Sep 17 20:49:50.821311 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:50.821290    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-94.ec2.internal\" not found"
Sep 17 20:49:50.842914 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.842889    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5358f381f605e5079b3247c4599e574-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal\" (UID: \"c5358f381f605e5079b3247c4599e574\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal"
Sep 17 20:49:50.843022 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.842922    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5358f381f605e5079b3247c4599e574-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal\" (UID: \"c5358f381f605e5079b3247c4599e574\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal"
Sep 17 20:49:50.843022 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.842952    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/26576e07a04b23124eb74ce2ed7324c3-config\") pod \"kube-apiserver-proxy-ip-10-0-131-94.ec2.internal\" (UID: \"26576e07a04b23124eb74ce2ed7324c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.ec2.internal"
Sep 17 20:49:50.922139 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:50.922111    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-94.ec2.internal\" not found"
Sep 17 20:49:50.943481 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.943459    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5358f381f605e5079b3247c4599e574-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal\" (UID: \"c5358f381f605e5079b3247c4599e574\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal"
Sep 17 20:49:50.943573 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.943493    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5358f381f605e5079b3247c4599e574-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal\" (UID: \"c5358f381f605e5079b3247c4599e574\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal"
Sep 17 20:49:50.943573 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.943518    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/26576e07a04b23124eb74ce2ed7324c3-config\") pod \"kube-apiserver-proxy-ip-10-0-131-94.ec2.internal\" (UID: \"26576e07a04b23124eb74ce2ed7324c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.ec2.internal"
Sep 17 20:49:50.943573 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.943544    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5358f381f605e5079b3247c4599e574-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal\" (UID: \"c5358f381f605e5079b3247c4599e574\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal"
Sep 17 20:49:50.943665 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.943563    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5358f381f605e5079b3247c4599e574-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal\" (UID: \"c5358f381f605e5079b3247c4599e574\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal"
Sep 17 20:49:50.943665 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:50.943565    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/26576e07a04b23124eb74ce2ed7324c3-config\") pod \"kube-apiserver-proxy-ip-10-0-131-94.ec2.internal\" (UID: \"26576e07a04b23124eb74ce2ed7324c3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.ec2.internal"
Sep 17 20:49:51.022886 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:51.022816    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-94.ec2.internal\" not found"
Sep 17 20:49:51.105305 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:51.105265    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal"
Sep 17 20:49:51.109352 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:51.109325    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.ec2.internal"
Sep 17 20:49:51.123391 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:51.123354    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-94.ec2.internal\" not found"
Sep 17 20:49:51.223971 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:51.223919    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-94.ec2.internal\" not found"
Sep 17 20:49:51.324476 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:51.324398    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-94.ec2.internal\" not found"
Sep 17 20:49:51.409864 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:51.409832    2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:51.424735 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:51.424711    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-94.ec2.internal\" not found"
Sep 17 20:49:51.458195 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:51.458165    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 20:49:51.458667 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:51.458290    2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 20:49:51.458667 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:51.458322    2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 20:49:51.458667 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:51.458329    2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 20:49:51.525815 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:51.525648    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-94.ec2.internal\" not found"
Sep 17 20:49:51.539850 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:51.539819    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 20:44:50 +0000 UTC" deadline="2028-06-08 11:34:04.325692416 +0000 UTC"
Sep 17 20:49:51.539850 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:51.539846    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15110h44m12.785848825s"
Sep 17 20:49:51.550265 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:51.550245    2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:51.640102 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:51.640075    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal"
Sep 17 20:49:51.651678 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:51.651662    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 20:49:51.652825 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:51.652812    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.ec2.internal"
Sep 17 20:49:51.661479 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:51.661464    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 20:49:51.728690 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:51.728647    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26576e07a04b23124eb74ce2ed7324c3.slice/crio-5f2bb35f609a1e5a1994991b39b99e741dfecfbe5a5ef077b80179ba7f10637f WatchSource:0}: Error finding container 5f2bb35f609a1e5a1994991b39b99e741dfecfbe5a5ef077b80179ba7f10637f: Status 404 returned error can't find the container with id 5f2bb35f609a1e5a1994991b39b99e741dfecfbe5a5ef077b80179ba7f10637f
Sep 17 20:49:51.728943 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:51.728911    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5358f381f605e5079b3247c4599e574.slice/crio-4fd373ee30c3259539c45a3331730166236b654ee6c1566846b8c69be3849de4 WatchSource:0}: Error finding container 4fd373ee30c3259539c45a3331730166236b654ee6c1566846b8c69be3849de4: Status 404 returned error can't find the container with id 4fd373ee30c3259539c45a3331730166236b654ee6c1566846b8c69be3849de4
Sep 17 20:49:51.733694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:51.733676    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 20:49:52.491029 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.490999    2571 apiserver.go:52] "Watching apiserver"
Sep 17 20:49:52.497784 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.497755    2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:52.498792 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.498769    2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 20:49:52.499829 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.499769    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-6742m","openshift-network-operator/iptables-alerter-mw75s","kube-system/global-pull-secret-syncer-2h2gl","kube-system/konnectivity-agent-nm7bs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal","openshift-multus/multus-9dxv5","openshift-network-diagnostics/network-check-target-csv4n","openshift-ovn-kubernetes/ovnkube-node-8wlt5","kube-system/kube-apiserver-proxy-ip-10-0-131-94.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb","openshift-cluster-node-tuning-operator/tuned-zwvhn","openshift-dns/node-resolver-n779j","openshift-image-registry/node-ca-qmv8f","openshift-multus/multus-additional-cni-plugins-47dgq"]
Sep 17 20:49:52.502466 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.502443    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:49:52.502595 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:52.502569    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:49:52.504724 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.504699    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.506886 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.506864    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:49:52.506977 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:52.506941    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:49:52.508008 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.507652    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 20:49:52.508008 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.507712    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:49:52.508008 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.507884    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kcvn7\""
Sep 17 20:49:52.509039 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.509019    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nm7bs"
Sep 17 20:49:52.511330 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.511312    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.512047 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.512026    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 20:49:52.512144 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.512087    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4q72w\""
Sep 17 20:49:52.513688 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.512727    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 20:49:52.513688 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.513377    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.514214 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.514196    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 20:49:52.514880 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.514458    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 20:49:52.514880 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.514630    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 20:49:52.514880 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.514712    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 20:49:52.514880 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.514866    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-57w4k\""
Sep 17 20:49:52.516006 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.515987    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.516754 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.516733    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 20:49:52.516924 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.516905    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 20:49:52.517860 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.517841    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cqp4q\""
Sep 17 20:49:52.518643 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.518623    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qmv8f"
Sep 17 20:49:52.519035 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.519017    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 20:49:52.519189 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.519171    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qfkhw\""
Sep 17 20:49:52.519376 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.519359    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 20:49:52.519441 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.519052    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 20:49:52.519750 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.519736    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 20:49:52.519905 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.519746    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 20:49:52.520001 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.519796    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 20:49:52.521728 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.521711    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mw75s"
Sep 17 20:49:52.522219 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.522073    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 20:49:52.522426 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.522406    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 20:49:52.523206 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.523115    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 20:49:52.523290 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.523204    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-52rmj\""
Sep 17 20:49:52.524305 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.524287    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.524701 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.524684    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 20:49:52.524840 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.524824    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:49:52.525311 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.525138    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 20:49:52.526443 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.526425    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cb2gq\""
Sep 17 20:49:52.526691 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.526674    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 20:49:52.526808 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.526789    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 20:49:52.527063 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.527040    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 20:49:52.527336 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.527318    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v7f42\""
Sep 17 20:49:52.528046 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.527797    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:49:52.528046 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:52.527872    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:49:52.530824 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.530806    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n779j"
Sep 17 20:49:52.533638 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.533464    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 20:49:52.534190 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.534057    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fsgqd\""
Sep 17 20:49:52.534190 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.534148    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 20:49:52.540023 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.540001    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 20:49:52.540966 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.540949    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 20:49:52.551720 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.551696    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 20:49:52.551836 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.551811    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d934ceba-229e-4206-bb5f-31b8e8af019f-serviceca\") pod \"node-ca-qmv8f\" (UID: \"d934ceba-229e-4206-bb5f-31b8e8af019f\") " pod="openshift-image-registry/node-ca-qmv8f"
Sep 17 20:49:52.551906 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.551845    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-kubelet\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.551906 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.551879    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-log-socket\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.552004 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.551904    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-sysctl-conf\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.552004 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.551938    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-sys\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.552004 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.551967    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ada893c-8b3a-46bf-a871-060c70e9cf88-konnectivity-ca\") pod \"konnectivity-agent-nm7bs\" (UID: \"9ada893c-8b3a-46bf-a871-060c70e9cf88\") " pod="kube-system/konnectivity-agent-nm7bs"
Sep 17 20:49:52.552004 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.551992    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/89ae281b-8074-4ef8-a264-991b56d6e3ff-ovn-node-metrics-cert\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.552179 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552018    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-registration-dir\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.552179 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552033    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-modprobe-d\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.552179 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552048    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-cni-binary-copy\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.552179 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552072    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-run-k8s-cni-cncf-io\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.552179 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552086    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-var-lib-kubelet\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.552179 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552100    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16b58fb3-0631-4fc1-894a-1d1f7a48b468-tuning-conf-dir\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.552179 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552118    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-run-openvswitch\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.552179 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552141    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa88803e-75a6-46c7-b9b4-82e86bbf3540-host-slash\") pod \"iptables-alerter-mw75s\" (UID: \"fa88803e-75a6-46c7-b9b4-82e86bbf3540\") " pod="openshift-network-operator/iptables-alerter-mw75s"
Sep 17 20:49:52.552179 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552163    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-device-dir\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.552535 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552185    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-etc-selinux\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.552535 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552209    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8wvj\" (UniqueName: \"kubernetes.io/projected/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-kube-api-access-r8wvj\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.552535 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552229    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-run-systemd\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.552535 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552254    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.552535 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552277    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-sysctl-d\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.552535 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552303    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-var-lib-kubelet\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.552535 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552328    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz5tx\" (UniqueName: \"kubernetes.io/projected/d934ceba-229e-4206-bb5f-31b8e8af019f-kube-api-access-fz5tx\") pod \"node-ca-qmv8f\" (UID: \"d934ceba-229e-4206-bb5f-31b8e8af019f\") " pod="openshift-image-registry/node-ca-qmv8f"
Sep 17 20:49:52.552535 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552352    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-multus-daemon-config\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.552535 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552387    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-run-multus-certs\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.552535 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552436    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/16b58fb3-0631-4fc1-894a-1d1f7a48b468-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.552535 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552470    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-systemd-units\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.552535 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552521    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-var-lib-openvswitch\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.553092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552564    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.553092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552595    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-run\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.553092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552616    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csw2f\" (UniqueName: \"kubernetes.io/projected/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-kube-api-access-csw2f\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.553092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552638    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-slash\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.553092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552735    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/89ae281b-8074-4ef8-a264-991b56d6e3ff-ovnkube-config\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.553092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552768    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0cc2bef5-b9e2-4a70-9905-f425604d5f3c-hosts-file\") pod \"node-resolver-n779j\" (UID: \"0cc2bef5-b9e2-4a70-9905-f425604d5f3c\") " pod="openshift-dns/node-resolver-n779j"
Sep 17 20:49:52.553092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552806    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-host\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.553092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552862    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41b7701f-d7a6-4924-9da1-040cc385bd19-tmp\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.553092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552895    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-multus-conf-dir\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.553092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552920    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-cni-netd\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.553092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552953    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d934ceba-229e-4206-bb5f-31b8e8af019f-host\") pod \"node-ca-qmv8f\" (UID: \"d934ceba-229e-4206-bb5f-31b8e8af019f\") " pod="openshift-image-registry/node-ca-qmv8f"
Sep 17 20:49:52.553092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552976    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4vtx\" (UniqueName: \"kubernetes.io/projected/fa88803e-75a6-46c7-b9b4-82e86bbf3540-kube-api-access-r4vtx\") pod \"iptables-alerter-mw75s\" (UID: \"fa88803e-75a6-46c7-b9b4-82e86bbf3540\") " pod="openshift-network-operator/iptables-alerter-mw75s"
Sep 17 20:49:52.553092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.552999    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-sys-fs\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.553092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553021    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vjj5\" (UniqueName: \"kubernetes.io/projected/16b58fb3-0631-4fc1-894a-1d1f7a48b468-kube-api-access-9vjj5\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.553092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553044    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-cni-bin\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.553092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553071    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm6vq\" (UniqueName: \"kubernetes.io/projected/41b7701f-d7a6-4924-9da1-040cc385bd19-kube-api-access-xm6vq\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.553719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553103    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cfe78836-a877-45c6-9b50-c5a481497f71-dbus\") pod \"global-pull-secret-syncer-2h2gl\" (UID: \"cfe78836-a877-45c6-9b50-c5a481497f71\") " pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:49:52.553719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553156    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-multus-socket-dir-parent\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.553719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553195    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-run-netns\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.553719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553233    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.553719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553257    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-socket-dir\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.553719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553280    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16b58fb3-0631-4fc1-894a-1d1f7a48b468-cnibin\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.553719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553303    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/89ae281b-8074-4ef8-a264-991b56d6e3ff-ovnkube-script-lib\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.553719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553325    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-sysconfig\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.553719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553346    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cfe78836-a877-45c6-9b50-c5a481497f71-kubelet-config\") pod \"global-pull-secret-syncer-2h2gl\" (UID: \"cfe78836-a877-45c6-9b50-c5a481497f71\") " pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:49:52.553719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553372    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-system-cni-dir\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.553719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553402    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-var-lib-cni-bin\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.553719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553426    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-run-ovn\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.553719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553465    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs\") pod \"network-metrics-daemon-6742m\" (UID: \"d2215eda-bca2-49dd-9419-90f9736fdf41\") " pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:49:52.553719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553497    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-577br\" (UniqueName: \"kubernetes.io/projected/0cc2bef5-b9e2-4a70-9905-f425604d5f3c-kube-api-access-577br\") pod \"node-resolver-n779j\" (UID: \"0cc2bef5-b9e2-4a70-9905-f425604d5f3c\") " pod="openshift-dns/node-resolver-n779j"
Sep 17 20:49:52.553719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553520    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-systemd\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.553719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553544    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-run-netns\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.554254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553584    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-tuned\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.554254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553606    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa88803e-75a6-46c7-b9b4-82e86bbf3540-iptables-alerter-script\") pod \"iptables-alerter-mw75s\" (UID: \"fa88803e-75a6-46c7-b9b4-82e86bbf3540\") " pod="openshift-network-operator/iptables-alerter-mw75s"
Sep 17 20:49:52.554254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553633    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-etc-openvswitch\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.554254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553655    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-node-log\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.554254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553676    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-etc-kubernetes\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.554254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553696    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-multus-cni-dir\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.554254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553717    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-cnibin\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.554254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553740    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-var-lib-cni-multus\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.554254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553762    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16b58fb3-0631-4fc1-894a-1d1f7a48b468-system-cni-dir\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.554254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553784    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/16b58fb3-0631-4fc1-894a-1d1f7a48b468-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.554254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553806    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/89ae281b-8074-4ef8-a264-991b56d6e3ff-env-overrides\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.554254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553828    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret\") pod \"global-pull-secret-syncer-2h2gl\" (UID: \"cfe78836-a877-45c6-9b50-c5a481497f71\") " pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:49:52.554254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553854    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m9gk\" (UniqueName: \"kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk\") pod \"network-check-target-csv4n\" (UID: \"a47b9a17-f95a-4e22-9584-a457ebbefdaf\") " pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:49:52.554254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553876    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wdsv\" (UniqueName: \"kubernetes.io/projected/89ae281b-8074-4ef8-a264-991b56d6e3ff-kube-api-access-4wdsv\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.554254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553898    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-kubernetes\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.554254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553929    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-lib-modules\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.554853 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553951    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ada893c-8b3a-46bf-a871-060c70e9cf88-agent-certs\") pod \"konnectivity-agent-nm7bs\" (UID: \"9ada893c-8b3a-46bf-a871-060c70e9cf88\") " pod="kube-system/konnectivity-agent-nm7bs"
Sep 17 20:49:52.554853 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553972    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-hostroot\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.554853 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.553994    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16b58fb3-0631-4fc1-894a-1d1f7a48b468-cni-binary-copy\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.554853 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.554018    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzz9c\" (UniqueName: \"kubernetes.io/projected/d2215eda-bca2-49dd-9419-90f9736fdf41-kube-api-access-gzz9c\") pod \"network-metrics-daemon-6742m\" (UID: \"d2215eda-bca2-49dd-9419-90f9736fdf41\") " pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:49:52.554853 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.554057    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc2bef5-b9e2-4a70-9905-f425604d5f3c-tmp-dir\") pod \"node-resolver-n779j\" (UID: \"0cc2bef5-b9e2-4a70-9905-f425604d5f3c\") " pod="openshift-dns/node-resolver-n779j"
Sep 17 20:49:52.602613 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.602592    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zpqmn"
Sep 17 20:49:52.612185 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.612161    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zpqmn"
Sep 17 20:49:52.654920 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.654885    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cfe78836-a877-45c6-9b50-c5a481497f71-dbus\") pod \"global-pull-secret-syncer-2h2gl\" (UID: \"cfe78836-a877-45c6-9b50-c5a481497f71\") " pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:49:52.655113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.654931    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-multus-socket-dir-parent\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.655113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.654957    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-run-netns\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.655113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.654980    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.655113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655001    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-socket-dir\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.655113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655023    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16b58fb3-0631-4fc1-894a-1d1f7a48b468-cnibin\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.655113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655045    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/89ae281b-8074-4ef8-a264-991b56d6e3ff-ovnkube-script-lib\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.655113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655064    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-run-netns\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.655113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655087    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-sysconfig\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655123    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cfe78836-a877-45c6-9b50-c5a481497f71-dbus\") pod \"global-pull-secret-syncer-2h2gl\" (UID: \"cfe78836-a877-45c6-9b50-c5a481497f71\") " pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655127    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cfe78836-a877-45c6-9b50-c5a481497f71-kubelet-config\") pod \"global-pull-secret-syncer-2h2gl\" (UID: \"cfe78836-a877-45c6-9b50-c5a481497f71\") " pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655146    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-multus-socket-dir-parent\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655171    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-system-cni-dir\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655190    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cfe78836-a877-45c6-9b50-c5a481497f71-kubelet-config\") pod \"global-pull-secret-syncer-2h2gl\" (UID: \"cfe78836-a877-45c6-9b50-c5a481497f71\") " pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655196    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-var-lib-cni-bin\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655235    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-run-ovn\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655253    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-var-lib-cni-bin\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655262    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs\") pod \"network-metrics-daemon-6742m\" (UID: \"d2215eda-bca2-49dd-9419-90f9736fdf41\") " pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655296    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-577br\" (UniqueName: \"kubernetes.io/projected/0cc2bef5-b9e2-4a70-9905-f425604d5f3c-kube-api-access-577br\") pod \"node-resolver-n779j\" (UID: \"0cc2bef5-b9e2-4a70-9905-f425604d5f3c\") " pod="openshift-dns/node-resolver-n779j"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655309    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-system-cni-dir\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655320    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-systemd\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655348    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-run-netns\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655372    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-tuned\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655394    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa88803e-75a6-46c7-b9b4-82e86bbf3540-iptables-alerter-script\") pod \"iptables-alerter-mw75s\" (UID: \"fa88803e-75a6-46c7-b9b4-82e86bbf3540\") " pod="openshift-network-operator/iptables-alerter-mw75s"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655419    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-etc-openvswitch\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655442    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-node-log\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.655523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655473    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-etc-kubernetes\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655494    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-multus-cni-dir\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655516    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-cnibin\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655572    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-var-lib-cni-multus\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655601    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16b58fb3-0631-4fc1-894a-1d1f7a48b468-system-cni-dir\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655629    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/16b58fb3-0631-4fc1-894a-1d1f7a48b468-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655662    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/89ae281b-8074-4ef8-a264-991b56d6e3ff-env-overrides\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655693    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret\") pod \"global-pull-secret-syncer-2h2gl\" (UID: \"cfe78836-a877-45c6-9b50-c5a481497f71\") " pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655724    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9gk\" (UniqueName: \"kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk\") pod \"network-check-target-csv4n\" (UID: \"a47b9a17-f95a-4e22-9584-a457ebbefdaf\") " pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655755    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wdsv\" (UniqueName: \"kubernetes.io/projected/89ae281b-8074-4ef8-a264-991b56d6e3ff-kube-api-access-4wdsv\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655781    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-kubernetes\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655822    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-lib-modules\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655855    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ada893c-8b3a-46bf-a871-060c70e9cf88-agent-certs\") pod \"konnectivity-agent-nm7bs\" (UID: \"9ada893c-8b3a-46bf-a871-060c70e9cf88\") " pod="kube-system/konnectivity-agent-nm7bs"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655879    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-hostroot\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655908    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16b58fb3-0631-4fc1-894a-1d1f7a48b468-cni-binary-copy\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655935    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzz9c\" (UniqueName: \"kubernetes.io/projected/d2215eda-bca2-49dd-9419-90f9736fdf41-kube-api-access-gzz9c\") pod \"network-metrics-daemon-6742m\" (UID: \"d2215eda-bca2-49dd-9419-90f9736fdf41\") " pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655942    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.656401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.655962    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc2bef5-b9e2-4a70-9905-f425604d5f3c-tmp-dir\") pod \"node-resolver-n779j\" (UID: \"0cc2bef5-b9e2-4a70-9905-f425604d5f3c\") " pod="openshift-dns/node-resolver-n779j"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656018    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d934ceba-229e-4206-bb5f-31b8e8af019f-serviceca\") pod \"node-ca-qmv8f\" (UID: \"d934ceba-229e-4206-bb5f-31b8e8af019f\") " pod="openshift-image-registry/node-ca-qmv8f"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656063    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-socket-dir\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656108    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/16b58fb3-0631-4fc1-894a-1d1f7a48b468-cnibin\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656144    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-kubelet\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656170    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-log-socket\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656205    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-sysctl-conf\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656231    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-sys\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656266    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-run-ovn\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656268    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ada893c-8b3a-46bf-a871-060c70e9cf88-konnectivity-ca\") pod \"konnectivity-agent-nm7bs\" (UID: \"9ada893c-8b3a-46bf-a871-060c70e9cf88\") " pod="kube-system/konnectivity-agent-nm7bs"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656311    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/89ae281b-8074-4ef8-a264-991b56d6e3ff-ovn-node-metrics-cert\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656345    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-registration-dir\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656373    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-modprobe-d\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656396    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-cni-binary-copy\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656420    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-run-k8s-cni-cncf-io\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656444    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-var-lib-kubelet\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656469    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16b58fb3-0631-4fc1-894a-1d1f7a48b468-tuning-conf-dir\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.657237 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656502    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-run-openvswitch\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656527    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa88803e-75a6-46c7-b9b4-82e86bbf3540-host-slash\") pod \"iptables-alerter-mw75s\" (UID: \"fa88803e-75a6-46c7-b9b4-82e86bbf3540\") " pod="openshift-network-operator/iptables-alerter-mw75s"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:52.656655    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:52.656708    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656753    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-systemd\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656801    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-run-netns\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656816    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/89ae281b-8074-4ef8-a264-991b56d6e3ff-ovnkube-script-lib\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656819    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-kubernetes\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656834    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ada893c-8b3a-46bf-a871-060c70e9cf88-konnectivity-ca\") pod \"konnectivity-agent-nm7bs\" (UID: \"9ada893c-8b3a-46bf-a871-060c70e9cf88\") " pod="kube-system/konnectivity-agent-nm7bs"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.656930    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-sys\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657067    2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657099    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-kubelet\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657110    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc2bef5-b9e2-4a70-9905-f425604d5f3c-tmp-dir\") pod \"node-resolver-n779j\" (UID: \"0cc2bef5-b9e2-4a70-9905-f425604d5f3c\") " pod="openshift-dns/node-resolver-n779j"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657142    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/16b58fb3-0631-4fc1-894a-1d1f7a48b468-cni-binary-copy\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657163    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-log-socket\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657250    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-lib-modules\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657275    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-sysctl-conf\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657371    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-device-dir\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657376    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d934ceba-229e-4206-bb5f-31b8e8af019f-serviceca\") pod \"node-ca-qmv8f\" (UID: \"d934ceba-229e-4206-bb5f-31b8e8af019f\") " pod="openshift-image-registry/node-ca-qmv8f"
Sep 17 20:49:52.658127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657403    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-etc-selinux\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.659113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657438    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8wvj\" (UniqueName: \"kubernetes.io/projected/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-kube-api-access-r8wvj\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.659113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657442    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-hostroot\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.659113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657460    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-run-openvswitch\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.659113 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:52.657505    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret podName:cfe78836-a877-45c6-9b50-c5a481497f71 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:53.157464106 +0000 UTC m=+4.661941689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret") pod "global-pull-secret-syncer-2h2gl" (UID: "cfe78836-a877-45c6-9b50-c5a481497f71") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:52.659113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657511    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-run-systemd\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.659113 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:52.657526    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs podName:d2215eda-bca2-49dd-9419-90f9736fdf41 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:53.157515421 +0000 UTC m=+4.661993008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs") pod "network-metrics-daemon-6742m" (UID: "d2215eda-bca2-49dd-9419-90f9736fdf41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:52.659113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657570    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-run-k8s-cni-cncf-io\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.659113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657584    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-etc-selinux\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.659113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657608    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-var-lib-kubelet\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.659113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657607    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa88803e-75a6-46c7-b9b4-82e86bbf3540-host-slash\") pod \"iptables-alerter-mw75s\" (UID: \"fa88803e-75a6-46c7-b9b4-82e86bbf3540\") " pod="openshift-network-operator/iptables-alerter-mw75s"
Sep 17 20:49:52.659113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657464    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-run-systemd\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.659113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657651    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-device-dir\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.659113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657655    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.659113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657686    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-sysctl-d\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.659113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657712    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-var-lib-kubelet\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.659113 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657740    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz5tx\" (UniqueName: \"kubernetes.io/projected/d934ceba-229e-4206-bb5f-31b8e8af019f-kube-api-access-fz5tx\") pod \"node-ca-qmv8f\" (UID: \"d934ceba-229e-4206-bb5f-31b8e8af019f\") " pod="openshift-image-registry/node-ca-qmv8f"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657763    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/16b58fb3-0631-4fc1-894a-1d1f7a48b468-tuning-conf-dir\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657799    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657765    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-multus-daemon-config\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657815    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-cni-binary-copy\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657840    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-run-multus-certs\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657864    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-sysctl-d\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657867    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/16b58fb3-0631-4fc1-894a-1d1f7a48b468-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657872    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-registration-dir\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657896    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-systemd-units\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657929    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-systemd-units\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657928    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-var-lib-kubelet\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.657973    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-modprobe-d\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658005    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-var-lib-openvswitch\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658034    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658062    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-run\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658084    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csw2f\" (UniqueName: \"kubernetes.io/projected/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-kube-api-access-csw2f\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.660698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658108    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-slash\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658140    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/89ae281b-8074-4ef8-a264-991b56d6e3ff-ovnkube-config\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658144    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-run-multus-certs\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658162    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0cc2bef5-b9e2-4a70-9905-f425604d5f3c-hosts-file\") pod \"node-resolver-n779j\" (UID: \"0cc2bef5-b9e2-4a70-9905-f425604d5f3c\") " pod="openshift-dns/node-resolver-n779j"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658208    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-cnibin\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658211    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0cc2bef5-b9e2-4a70-9905-f425604d5f3c-hosts-file\") pod \"node-resolver-n779j\" (UID: \"0cc2bef5-b9e2-4a70-9905-f425604d5f3c\") " pod="openshift-dns/node-resolver-n779j"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658248    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-host\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658254    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-var-lib-openvswitch\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658275    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41b7701f-d7a6-4924-9da1-040cc385bd19-tmp\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658286    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658301    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-multus-conf-dir\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658322    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-run\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658328    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-cni-netd\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658349    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d934ceba-229e-4206-bb5f-31b8e8af019f-host\") pod \"node-ca-qmv8f\" (UID: \"d934ceba-229e-4206-bb5f-31b8e8af019f\") " pod="openshift-image-registry/node-ca-qmv8f"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658373    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-multus-daemon-config\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658377    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4vtx\" (UniqueName: \"kubernetes.io/projected/fa88803e-75a6-46c7-b9b4-82e86bbf3540-kube-api-access-r4vtx\") pod \"iptables-alerter-mw75s\" (UID: \"fa88803e-75a6-46c7-b9b4-82e86bbf3540\") " pod="openshift-network-operator/iptables-alerter-mw75s"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658403    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-sys-fs\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658457    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-sys-fs\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.661583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658819    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/16b58fb3-0631-4fc1-894a-1d1f7a48b468-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658894    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vjj5\" (UniqueName: \"kubernetes.io/projected/16b58fb3-0631-4fc1-894a-1d1f7a48b468-kube-api-access-9vjj5\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658923    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-cni-bin\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658941    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/89ae281b-8074-4ef8-a264-991b56d6e3ff-ovnkube-config\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.658949    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm6vq\" (UniqueName: \"kubernetes.io/projected/41b7701f-d7a6-4924-9da1-040cc385bd19-kube-api-access-xm6vq\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.659015    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa88803e-75a6-46c7-b9b4-82e86bbf3540-iptables-alerter-script\") pod \"iptables-alerter-mw75s\" (UID: \"fa88803e-75a6-46c7-b9b4-82e86bbf3540\") " pod="openshift-network-operator/iptables-alerter-mw75s"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.659089    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-slash\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.659153    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-etc-openvswitch\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.659195    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-cni-bin\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.659202    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-node-log\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.659247    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-etc-kubernetes\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.659304    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-multus-cni-dir\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.659314    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-sysconfig\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.659346    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41b7701f-d7a6-4924-9da1-040cc385bd19-host\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.659535    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89ae281b-8074-4ef8-a264-991b56d6e3ff-host-cni-netd\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.659678    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-multus-conf-dir\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.659753    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-host-var-lib-cni-multus\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.659785    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/16b58fb3-0631-4fc1-894a-1d1f7a48b468-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.662100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.659835    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/16b58fb3-0631-4fc1-894a-1d1f7a48b468-system-cni-dir\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.662631 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.659944    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d934ceba-229e-4206-bb5f-31b8e8af019f-host\") pod \"node-ca-qmv8f\" (UID: \"d934ceba-229e-4206-bb5f-31b8e8af019f\") " pod="openshift-image-registry/node-ca-qmv8f"
Sep 17 20:49:52.662631 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.660237    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/89ae281b-8074-4ef8-a264-991b56d6e3ff-env-overrides\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.668572 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.665972    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41b7701f-d7a6-4924-9da1-040cc385bd19-etc-tuned\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.668572 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.666348    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/89ae281b-8074-4ef8-a264-991b56d6e3ff-ovn-node-metrics-cert\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.668572 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.666729    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41b7701f-d7a6-4924-9da1-040cc385bd19-tmp\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.668572 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.667646    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ada893c-8b3a-46bf-a871-060c70e9cf88-agent-certs\") pod \"konnectivity-agent-nm7bs\" (UID: \"9ada893c-8b3a-46bf-a871-060c70e9cf88\") " pod="kube-system/konnectivity-agent-nm7bs"
Sep 17 20:49:52.671836 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.671644    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-577br\" (UniqueName: \"kubernetes.io/projected/0cc2bef5-b9e2-4a70-9905-f425604d5f3c-kube-api-access-577br\") pod \"node-resolver-n779j\" (UID: \"0cc2bef5-b9e2-4a70-9905-f425604d5f3c\") " pod="openshift-dns/node-resolver-n779j"
Sep 17 20:49:52.672870 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.672779    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.ec2.internal" event={"ID":"26576e07a04b23124eb74ce2ed7324c3","Type":"ContainerStarted","Data":"5f2bb35f609a1e5a1994991b39b99e741dfecfbe5a5ef077b80179ba7f10637f"}
Sep 17 20:49:52.673601 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.673577    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal" event={"ID":"c5358f381f605e5079b3247c4599e574","Type":"ContainerStarted","Data":"4fd373ee30c3259539c45a3331730166236b654ee6c1566846b8c69be3849de4"}
Sep 17 20:49:52.675154 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.675134    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wdsv\" (UniqueName: \"kubernetes.io/projected/89ae281b-8074-4ef8-a264-991b56d6e3ff-kube-api-access-4wdsv\") pod \"ovnkube-node-8wlt5\" (UID: \"89ae281b-8074-4ef8-a264-991b56d6e3ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.677108 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.677086    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4vtx\" (UniqueName: \"kubernetes.io/projected/fa88803e-75a6-46c7-b9b4-82e86bbf3540-kube-api-access-r4vtx\") pod \"iptables-alerter-mw75s\" (UID: \"fa88803e-75a6-46c7-b9b4-82e86bbf3540\") " pod="openshift-network-operator/iptables-alerter-mw75s"
Sep 17 20:49:52.678992 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.678968    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz5tx\" (UniqueName: \"kubernetes.io/projected/d934ceba-229e-4206-bb5f-31b8e8af019f-kube-api-access-fz5tx\") pod \"node-ca-qmv8f\" (UID: \"d934ceba-229e-4206-bb5f-31b8e8af019f\") " pod="openshift-image-registry/node-ca-qmv8f"
Sep 17 20:49:52.679625 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.679602    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzz9c\" (UniqueName: \"kubernetes.io/projected/d2215eda-bca2-49dd-9419-90f9736fdf41-kube-api-access-gzz9c\") pod \"network-metrics-daemon-6742m\" (UID: \"d2215eda-bca2-49dd-9419-90f9736fdf41\") " pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:49:52.680444 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.680420    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8wvj\" (UniqueName: \"kubernetes.io/projected/1dcc7d98-518b-43c2-85d3-96c7e7af9dde-kube-api-access-r8wvj\") pod \"aws-ebs-csi-driver-node-qpldb\" (UID: \"1dcc7d98-518b-43c2-85d3-96c7e7af9dde\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.680894 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:52.680877    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:52.680985 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:52.680901    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:52.680985 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:52.680914    2571 projected.go:196] Error preparing data for projected volume kube-api-access-7m9gk for pod openshift-network-diagnostics/network-check-target-csv4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:52.680985 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:52.680979    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk podName:a47b9a17-f95a-4e22-9584-a457ebbefdaf nodeName:}" failed. No retries permitted until 2026-09-17 20:49:53.180962068 +0000 UTC m=+4.685439661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7m9gk" (UniqueName: "kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk") pod "network-check-target-csv4n" (UID: "a47b9a17-f95a-4e22-9584-a457ebbefdaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:52.686123 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.686098    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vjj5\" (UniqueName: \"kubernetes.io/projected/16b58fb3-0631-4fc1-894a-1d1f7a48b468-kube-api-access-9vjj5\") pod \"multus-additional-cni-plugins-47dgq\" (UID: \"16b58fb3-0631-4fc1-894a-1d1f7a48b468\") " pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.686720 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.686701    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm6vq\" (UniqueName: \"kubernetes.io/projected/41b7701f-d7a6-4924-9da1-040cc385bd19-kube-api-access-xm6vq\") pod \"tuned-zwvhn\" (UID: \"41b7701f-d7a6-4924-9da1-040cc385bd19\") " pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.686898 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.686877    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csw2f\" (UniqueName: \"kubernetes.io/projected/bf6c96de-21ca-45f6-b2f0-8b9b3b65411b-kube-api-access-csw2f\") pod \"multus-9dxv5\" (UID: \"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b\") " pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.817889 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.817859    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zwvhn"
Sep 17 20:49:52.827099 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.827075    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nm7bs"
Sep 17 20:49:52.838813 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.838792    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-47dgq"
Sep 17 20:49:52.844691 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.844671    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9dxv5"
Sep 17 20:49:52.860194 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.860174    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:49:52.868859 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.868840    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qmv8f"
Sep 17 20:49:52.873441 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.873419    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:52.876598 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.876577    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mw75s"
Sep 17 20:49:52.885136 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.885119    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb"
Sep 17 20:49:52.891688 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:52.891664    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n779j"
Sep 17 20:49:53.162270 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:53.162182    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs\") pod \"network-metrics-daemon-6742m\" (UID: \"d2215eda-bca2-49dd-9419-90f9736fdf41\") " pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:49:53.162270 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:53.162221    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret\") pod \"global-pull-secret-syncer-2h2gl\" (UID: \"cfe78836-a877-45c6-9b50-c5a481497f71\") " pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:49:53.162485 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:53.162343    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:53.162485 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:53.162412    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs podName:d2215eda-bca2-49dd-9419-90f9736fdf41 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:54.162391204 +0000 UTC m=+5.666868795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs") pod "network-metrics-daemon-6742m" (UID: "d2215eda-bca2-49dd-9419-90f9736fdf41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:53.162485 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:53.162346    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:53.162485 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:53.162465    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret podName:cfe78836-a877-45c6-9b50-c5a481497f71 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:54.162454207 +0000 UTC m=+5.666931782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret") pod "global-pull-secret-syncer-2h2gl" (UID: "cfe78836-a877-45c6-9b50-c5a481497f71") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:53.263134 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:53.263094    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9gk\" (UniqueName: \"kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk\") pod \"network-check-target-csv4n\" (UID: \"a47b9a17-f95a-4e22-9584-a457ebbefdaf\") " pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:49:53.263303 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:53.263264    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:53.263303 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:53.263294    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:53.263404 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:53.263308    2571 projected.go:196] Error preparing data for projected volume kube-api-access-7m9gk for pod openshift-network-diagnostics/network-check-target-csv4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:53.263404 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:53.263372    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk podName:a47b9a17-f95a-4e22-9584-a457ebbefdaf nodeName:}" failed. No retries permitted until 2026-09-17 20:49:54.263352409 +0000 UTC m=+5.767830003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7m9gk" (UniqueName: "kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk") pod "network-check-target-csv4n" (UID: "a47b9a17-f95a-4e22-9584-a457ebbefdaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:53.615013 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:53.614970    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 20:44:52 +0000 UTC" deadline="2028-05-05 13:43:14.809166981 +0000 UTC"
Sep 17 20:49:53.615013 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:53.615005    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14296h53m21.19416526s"
Sep 17 20:49:53.645031 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:53.644942    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ada893c_8b3a_46bf_a871_060c70e9cf88.slice/crio-441d8ee7b88925e90b93d4de1ba332461df35c137f511c69eb24fb5be217c7c4 WatchSource:0}: Error finding container 441d8ee7b88925e90b93d4de1ba332461df35c137f511c69eb24fb5be217c7c4: Status 404 returned error can't find the container with id 441d8ee7b88925e90b93d4de1ba332461df35c137f511c69eb24fb5be217c7c4
Sep 17 20:49:53.646303 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:53.645969    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcc7d98_518b_43c2_85d3_96c7e7af9dde.slice/crio-5a87900bbfb523621bbe9857e6cfcaa5dad033cdf41d92bf470ec6b2e7483a20 WatchSource:0}: Error finding container 5a87900bbfb523621bbe9857e6cfcaa5dad033cdf41d92bf470ec6b2e7483a20: Status 404 returned error can't find the container with id 5a87900bbfb523621bbe9857e6cfcaa5dad033cdf41d92bf470ec6b2e7483a20
Sep 17 20:49:53.647773 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:53.647751    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ae281b_8074_4ef8_a264_991b56d6e3ff.slice/crio-00b4e1303f062323c3776a04d927c645778eebebdbde14193c806c433f92dc3b WatchSource:0}: Error finding container 00b4e1303f062323c3776a04d927c645778eebebdbde14193c806c433f92dc3b: Status 404 returned error can't find the container with id 00b4e1303f062323c3776a04d927c645778eebebdbde14193c806c433f92dc3b
Sep 17 20:49:53.651675 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:53.651647    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b7701f_d7a6_4924_9da1_040cc385bd19.slice/crio-e1731abedaee7001b032c99604ac4810213b03a4abbddfd40e6344e710030476 WatchSource:0}: Error finding container e1731abedaee7001b032c99604ac4810213b03a4abbddfd40e6344e710030476: Status 404 returned error can't find the container with id e1731abedaee7001b032c99604ac4810213b03a4abbddfd40e6344e710030476
Sep 17 20:49:53.652498 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:53.652474    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6c96de_21ca_45f6_b2f0_8b9b3b65411b.slice/crio-aa6206e44256445d60004b8560ed58cbb2675cafce89cab45b2e1781588a7ccd WatchSource:0}: Error finding container aa6206e44256445d60004b8560ed58cbb2675cafce89cab45b2e1781588a7ccd: Status 404 returned error can't find the container with id aa6206e44256445d60004b8560ed58cbb2675cafce89cab45b2e1781588a7ccd
Sep 17 20:49:53.653626 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:53.653602    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa88803e_75a6_46c7_b9b4_82e86bbf3540.slice/crio-195b1d344af13b43ef5aaf7dcc59948d36cb400d1d8f1e14aa30d8c4ecccf13c WatchSource:0}: Error finding container 195b1d344af13b43ef5aaf7dcc59948d36cb400d1d8f1e14aa30d8c4ecccf13c: Status 404 returned error can't find the container with id 195b1d344af13b43ef5aaf7dcc59948d36cb400d1d8f1e14aa30d8c4ecccf13c
Sep 17 20:49:53.655734 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:49:53.655530    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc2bef5_b9e2_4a70_9905_f425604d5f3c.slice/crio-6f23dcd70888fa3b63a3c9f608c508cec23dd20dfcd624839e6c3edeca951596 WatchSource:0}: Error finding container 6f23dcd70888fa3b63a3c9f608c508cec23dd20dfcd624839e6c3edeca951596: Status 404 returned error can't find the container with id 6f23dcd70888fa3b63a3c9f608c508cec23dd20dfcd624839e6c3edeca951596
Sep 17 20:49:53.664433 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:53.664393    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:49:53.664563 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:53.664524    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:49:53.680517 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:53.680493    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n779j" event={"ID":"0cc2bef5-b9e2-4a70-9905-f425604d5f3c","Type":"ContainerStarted","Data":"6f23dcd70888fa3b63a3c9f608c508cec23dd20dfcd624839e6c3edeca951596"}
Sep 17 20:49:53.683334 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:53.683308    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9dxv5" event={"ID":"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b","Type":"ContainerStarted","Data":"aa6206e44256445d60004b8560ed58cbb2675cafce89cab45b2e1781588a7ccd"}
Sep 17 20:49:53.685294 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:53.685272    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nm7bs" event={"ID":"9ada893c-8b3a-46bf-a871-060c70e9cf88","Type":"ContainerStarted","Data":"441d8ee7b88925e90b93d4de1ba332461df35c137f511c69eb24fb5be217c7c4"}
Sep 17 20:49:53.689729 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:53.689703    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47dgq" event={"ID":"16b58fb3-0631-4fc1-894a-1d1f7a48b468","Type":"ContainerStarted","Data":"7bce4b20d89982b3ae8926233f8c1fee773a3809863e2c738c992d9cccc003d5"}
Sep 17 20:49:53.692375 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:53.692352    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qmv8f" event={"ID":"d934ceba-229e-4206-bb5f-31b8e8af019f","Type":"ContainerStarted","Data":"b50e63677cebcc768d1750e0d362c3f4adbbcb412a4018184d36e03ad7db618c"}
Sep 17 20:49:53.695229 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:53.695194    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mw75s" event={"ID":"fa88803e-75a6-46c7-b9b4-82e86bbf3540","Type":"ContainerStarted","Data":"195b1d344af13b43ef5aaf7dcc59948d36cb400d1d8f1e14aa30d8c4ecccf13c"}
Sep 17 20:49:53.696319 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:53.696300    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zwvhn" event={"ID":"41b7701f-d7a6-4924-9da1-040cc385bd19","Type":"ContainerStarted","Data":"e1731abedaee7001b032c99604ac4810213b03a4abbddfd40e6344e710030476"}
Sep 17 20:49:53.697498 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:53.697472    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5" event={"ID":"89ae281b-8074-4ef8-a264-991b56d6e3ff","Type":"ContainerStarted","Data":"00b4e1303f062323c3776a04d927c645778eebebdbde14193c806c433f92dc3b"}
Sep 17 20:49:53.698642 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:53.698620    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb" event={"ID":"1dcc7d98-518b-43c2-85d3-96c7e7af9dde","Type":"ContainerStarted","Data":"5a87900bbfb523621bbe9857e6cfcaa5dad033cdf41d92bf470ec6b2e7483a20"}
Sep 17 20:49:54.169818 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:54.169224    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs\") pod \"network-metrics-daemon-6742m\" (UID: \"d2215eda-bca2-49dd-9419-90f9736fdf41\") " pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:49:54.169818 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:54.169505    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret\") pod \"global-pull-secret-syncer-2h2gl\" (UID: \"cfe78836-a877-45c6-9b50-c5a481497f71\") " pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:49:54.169818 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:54.169585    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:54.169818 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:54.169631    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:54.169818 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:54.169665    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs podName:d2215eda-bca2-49dd-9419-90f9736fdf41 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:56.169643992 +0000 UTC m=+7.674121585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs") pod "network-metrics-daemon-6742m" (UID: "d2215eda-bca2-49dd-9419-90f9736fdf41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:54.169818 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:54.169687    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret podName:cfe78836-a877-45c6-9b50-c5a481497f71 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:56.169672331 +0000 UTC m=+7.674149919 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret") pod "global-pull-secret-syncer-2h2gl" (UID: "cfe78836-a877-45c6-9b50-c5a481497f71") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:54.273090 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:54.270506    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9gk\" (UniqueName: \"kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk\") pod \"network-check-target-csv4n\" (UID: \"a47b9a17-f95a-4e22-9584-a457ebbefdaf\") " pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:49:54.273090 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:54.270746    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:54.273090 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:54.270769    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:54.273090 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:54.270781    2571 projected.go:196] Error preparing data for projected volume kube-api-access-7m9gk for pod openshift-network-diagnostics/network-check-target-csv4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:54.273090 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:54.270845    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk podName:a47b9a17-f95a-4e22-9584-a457ebbefdaf nodeName:}" failed. No retries permitted until 2026-09-17 20:49:56.270826231 +0000 UTC m=+7.775303828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7m9gk" (UniqueName: "kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk") pod "network-check-target-csv4n" (UID: "a47b9a17-f95a-4e22-9584-a457ebbefdaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:54.535727 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:54.535539    2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:54.616141 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:54.616075    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 20:44:52 +0000 UTC" deadline="2028-04-07 14:26:09.74840365 +0000 UTC"
Sep 17 20:49:54.616141 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:54.616139    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13625h36m15.132269753s"
Sep 17 20:49:54.667124 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:54.667088    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:49:54.667297 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:54.667221    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:49:54.667637 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:54.667599    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:49:54.667731 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:54.667695    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:49:54.719570 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:54.715509    2571 generic.go:356] "Generic (PLEG): container finished" podID="c5358f381f605e5079b3247c4599e574" containerID="e6fe3d0727c7569c4910345973285f01c23fc7bfb5df9b49fa05bad6df8ad75d" exitCode=0
Sep 17 20:49:54.719570 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:54.715631    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal" event={"ID":"c5358f381f605e5079b3247c4599e574","Type":"ContainerDied","Data":"e6fe3d0727c7569c4910345973285f01c23fc7bfb5df9b49fa05bad6df8ad75d"}
Sep 17 20:49:54.725491 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:54.725453    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.ec2.internal" event={"ID":"26576e07a04b23124eb74ce2ed7324c3","Type":"ContainerStarted","Data":"a1cd94cd7290e324548b568c2e3e87cb1ec22a65161d6a6d0742e09054830829"}
Sep 17 20:49:54.772225 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:54.772164    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.ec2.internal" podStartSLOduration=3.772140333 podStartE2EDuration="3.772140333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:54.769305006 +0000 UTC m=+6.273782631" watchObservedRunningTime="2026-09-17 20:49:54.772140333 +0000 UTC m=+6.276617930"
Sep 17 20:49:55.664167 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:55.664131    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:49:55.664599 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:55.664278    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:49:55.745265 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:55.745169    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal" event={"ID":"c5358f381f605e5079b3247c4599e574","Type":"ContainerStarted","Data":"6b2bbbe9c322aeffb58d9227a3934567a0af6d1311e4c7c8e2b63826083e2735"}
Sep 17 20:49:56.189127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:56.189088    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret\") pod \"global-pull-secret-syncer-2h2gl\" (UID: \"cfe78836-a877-45c6-9b50-c5a481497f71\") " pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:49:56.189321 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:56.189188    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs\") pod \"network-metrics-daemon-6742m\" (UID: \"d2215eda-bca2-49dd-9419-90f9736fdf41\") " pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:49:56.189387 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:56.189329    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:56.189441 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:56.189393    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs podName:d2215eda-bca2-49dd-9419-90f9736fdf41 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:00.189373105 +0000 UTC m=+11.693850702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs") pod "network-metrics-daemon-6742m" (UID: "d2215eda-bca2-49dd-9419-90f9736fdf41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:56.189844 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:56.189819    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:56.189941 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:56.189878    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret podName:cfe78836-a877-45c6-9b50-c5a481497f71 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:00.189863699 +0000 UTC m=+11.694341287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret") pod "global-pull-secret-syncer-2h2gl" (UID: "cfe78836-a877-45c6-9b50-c5a481497f71") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:56.289693 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:56.289649    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9gk\" (UniqueName: \"kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk\") pod \"network-check-target-csv4n\" (UID: \"a47b9a17-f95a-4e22-9584-a457ebbefdaf\") " pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:49:56.289869 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:56.289824    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:56.289869 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:56.289843    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:56.289869 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:56.289858    2571 projected.go:196] Error preparing data for projected volume kube-api-access-7m9gk for pod openshift-network-diagnostics/network-check-target-csv4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:56.290039 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:56.289921    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk podName:a47b9a17-f95a-4e22-9584-a457ebbefdaf nodeName:}" failed. No retries permitted until 2026-09-17 20:50:00.289902904 +0000 UTC m=+11.794380501 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7m9gk" (UniqueName: "kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk") pod "network-check-target-csv4n" (UID: "a47b9a17-f95a-4e22-9584-a457ebbefdaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:56.665607 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:56.665570    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:49:56.666078 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:56.665714    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:49:56.666148 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:56.666109    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:49:56.666324 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:56.666201    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:49:57.663742 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:57.663706    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:49:57.663923 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:57.663851    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:49:58.664486 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:58.664448    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:49:58.664974 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:58.664618    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:49:58.665043 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:58.665010    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:49:58.665147 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:58.665108    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:49:59.665100 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:49:59.664605    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:49:59.665100 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:49:59.664751    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:50:00.227114 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:00.226352    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs\") pod \"network-metrics-daemon-6742m\" (UID: \"d2215eda-bca2-49dd-9419-90f9736fdf41\") " pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:00.227114 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:00.226411    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret\") pod \"global-pull-secret-syncer-2h2gl\" (UID: \"cfe78836-a877-45c6-9b50-c5a481497f71\") " pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:00.227114 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:00.226584    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:50:00.227114 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:00.226650    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret podName:cfe78836-a877-45c6-9b50-c5a481497f71 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:08.226629797 +0000 UTC m=+19.731107439 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret") pod "global-pull-secret-syncer-2h2gl" (UID: "cfe78836-a877-45c6-9b50-c5a481497f71") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:50:00.227114 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:00.227034    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:50:00.227114 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:00.227081    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs podName:d2215eda-bca2-49dd-9419-90f9736fdf41 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:08.227066307 +0000 UTC m=+19.731543895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs") pod "network-metrics-daemon-6742m" (UID: "d2215eda-bca2-49dd-9419-90f9736fdf41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:50:00.327004 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:00.326967    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9gk\" (UniqueName: \"kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk\") pod \"network-check-target-csv4n\" (UID: \"a47b9a17-f95a-4e22-9584-a457ebbefdaf\") " pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:00.327203 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:00.327136    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:50:00.327203 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:00.327159    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:50:00.327203 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:00.327172    2571 projected.go:196] Error preparing data for projected volume kube-api-access-7m9gk for pod openshift-network-diagnostics/network-check-target-csv4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:50:00.327373 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:00.327224    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk podName:a47b9a17-f95a-4e22-9584-a457ebbefdaf nodeName:}" failed. No retries permitted until 2026-09-17 20:50:08.327211151 +0000 UTC m=+19.831688724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7m9gk" (UniqueName: "kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk") pod "network-check-target-csv4n" (UID: "a47b9a17-f95a-4e22-9584-a457ebbefdaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:50:00.664977 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:00.664942    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:00.665127 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:00.665063    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:50:00.665634 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:00.665612    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:00.665754 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:00.665726    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:50:01.664584 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:01.664529    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:01.664873 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:01.664666    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:50:02.663790 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:02.663752    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:02.663790 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:02.663780    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:02.664253 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:02.663885    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:50:02.664253 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:02.664003    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:50:03.663753 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:03.663714    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:03.663944 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:03.663844    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:50:04.664211 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:04.664170    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:04.664646 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:04.664319    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:50:04.664646 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:04.664358    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:04.664646 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:04.664501    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:50:05.663983 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:05.663943    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:05.664152 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:05.664066    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:50:06.664235 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:06.664203    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:06.664697 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:06.664203    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:06.664697 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:06.664341    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:50:06.664697 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:06.664411    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:50:07.664487 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:07.664458    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:07.664891 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:07.664580    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:50:08.286576 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:08.286524    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs\") pod \"network-metrics-daemon-6742m\" (UID: \"d2215eda-bca2-49dd-9419-90f9736fdf41\") " pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:08.286576 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:08.286593    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret\") pod \"global-pull-secret-syncer-2h2gl\" (UID: \"cfe78836-a877-45c6-9b50-c5a481497f71\") " pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:08.286836 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:08.286690    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:50:08.286836 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:08.286690    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:50:08.286836 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:08.286749    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs podName:d2215eda-bca2-49dd-9419-90f9736fdf41 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:24.286730547 +0000 UTC m=+35.791208134 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs") pod "network-metrics-daemon-6742m" (UID: "d2215eda-bca2-49dd-9419-90f9736fdf41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:50:08.286836 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:08.286765    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret podName:cfe78836-a877-45c6-9b50-c5a481497f71 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:24.286757525 +0000 UTC m=+35.791235101 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret") pod "global-pull-secret-syncer-2h2gl" (UID: "cfe78836-a877-45c6-9b50-c5a481497f71") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:50:08.387934 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:08.387893    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9gk\" (UniqueName: \"kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk\") pod \"network-check-target-csv4n\" (UID: \"a47b9a17-f95a-4e22-9584-a457ebbefdaf\") " pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:08.388129 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:08.388108    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:50:08.388188 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:08.388134    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:50:08.388188 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:08.388144    2571 projected.go:196] Error preparing data for projected volume kube-api-access-7m9gk for pod openshift-network-diagnostics/network-check-target-csv4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:50:08.388299 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:08.388207    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk podName:a47b9a17-f95a-4e22-9584-a457ebbefdaf nodeName:}" failed. No retries permitted until 2026-09-17 20:50:24.388190216 +0000 UTC m=+35.892667817 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7m9gk" (UniqueName: "kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk") pod "network-check-target-csv4n" (UID: "a47b9a17-f95a-4e22-9584-a457ebbefdaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:50:08.664804 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:08.664771    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:08.665236 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:08.664898    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:50:08.665405 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:08.665386    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:08.665521 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:08.665502    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:50:09.664376 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:09.664338    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:09.664567 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:09.664484    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:50:10.665049 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:10.665000    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:10.665437 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:10.665118    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:50:10.665437 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:10.665174    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:10.665437 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:10.665300    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:50:11.664178 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.664014    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:11.664345 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:11.664291    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:50:11.784969 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.784936    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qmv8f" event={"ID":"d934ceba-229e-4206-bb5f-31b8e8af019f","Type":"ContainerStarted","Data":"8ce6c7dcedea5ec49fd86e1a1158057a92d7db695343a35d636da92c019d9ca3"}
Sep 17 20:50:11.786581 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.786531    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zwvhn" event={"ID":"41b7701f-d7a6-4924-9da1-040cc385bd19","Type":"ContainerStarted","Data":"619fb68eeb7317a9a8697107508da59b3d51ca1dc5e69f3b4090b01db5762f9a"}
Sep 17 20:50:11.789586 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.789562    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5" event={"ID":"89ae281b-8074-4ef8-a264-991b56d6e3ff","Type":"ContainerStarted","Data":"df2f2dc50f97678ca15e7daf55dd4145282b09b3c392e3625b1749ac061cb984"}
Sep 17 20:50:11.789695 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.789639    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5" event={"ID":"89ae281b-8074-4ef8-a264-991b56d6e3ff","Type":"ContainerStarted","Data":"44ca545dc109d044a7a896255925d7e650eb2d4013aa6cf9278223ac7404bc8a"}
Sep 17 20:50:11.789695 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.789690    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5" event={"ID":"89ae281b-8074-4ef8-a264-991b56d6e3ff","Type":"ContainerStarted","Data":"2f600a735b2a5b9109efa884c4eb3b47f365ffc33a9ae418b34e796ba21fee22"}
Sep 17 20:50:11.791092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.791071    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb" event={"ID":"1dcc7d98-518b-43c2-85d3-96c7e7af9dde","Type":"ContainerStarted","Data":"ba5a233a88b41d341b809c2ef369187f6cc6d9577985f07495545b293206b66d"}
Sep 17 20:50:11.792582 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.792516    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n779j" event={"ID":"0cc2bef5-b9e2-4a70-9905-f425604d5f3c","Type":"ContainerStarted","Data":"9141ccaf93d0002a3d740ec545612403ec5f6e153edf40012312d078c7016870"}
Sep 17 20:50:11.794242 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.794223    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9dxv5" event={"ID":"bf6c96de-21ca-45f6-b2f0-8b9b3b65411b","Type":"ContainerStarted","Data":"7d165b8fd531946303fcb9ac763f609756b522aad402f7b45ce6f91086430342"}
Sep 17 20:50:11.795862 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.795842    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nm7bs" event={"ID":"9ada893c-8b3a-46bf-a871-060c70e9cf88","Type":"ContainerStarted","Data":"904ff0c4482a79772f104a2d87c64d4fa97512605d4709625df478dc2a09d59d"}
Sep 17 20:50:11.797567 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.797530    2571 generic.go:356] "Generic (PLEG): container finished" podID="16b58fb3-0631-4fc1-894a-1d1f7a48b468" containerID="fcaf13594b02a318b2926a6285cc1cb83bbe1ca6333d561be23f28aebcb13b65" exitCode=0
Sep 17 20:50:11.797645 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.797586    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47dgq" event={"ID":"16b58fb3-0631-4fc1-894a-1d1f7a48b468","Type":"ContainerDied","Data":"fcaf13594b02a318b2926a6285cc1cb83bbe1ca6333d561be23f28aebcb13b65"}
Sep 17 20:50:11.800039 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.799993    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.ec2.internal" podStartSLOduration=20.799978082 podStartE2EDuration="20.799978082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:55.769056438 +0000 UTC m=+7.273534036" watchObservedRunningTime="2026-09-17 20:50:11.799978082 +0000 UTC m=+23.304455679"
Sep 17 20:50:11.800678 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.800641    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qmv8f" podStartSLOduration=4.115557621 podStartE2EDuration="21.800628361s" totalImagesPullingTime="17.68507074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:11.799732815 +0000 UTC m=+23.304210412" watchObservedRunningTime="2026-09-17 20:50:11.800628361 +0000 UTC m=+23.305105958"
Sep 17 20:50:11.812953 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.812912    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nm7bs" podStartSLOduration=8.034426956 podStartE2EDuration="21.812898221s" totalImagesPullingTime="13.778471265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:11.812617371 +0000 UTC m=+23.317094968" watchObservedRunningTime="2026-09-17 20:50:11.812898221 +0000 UTC m=+23.317375848"
Sep 17 20:50:11.853923 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.853882    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n779j" podStartSLOduration=4.167046161 podStartE2EDuration="21.853867591s" totalImagesPullingTime="17.68682143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:11.853604818 +0000 UTC m=+23.358082406" watchObservedRunningTime="2026-09-17 20:50:11.853867591 +0000 UTC m=+23.358345186"
Sep 17 20:50:11.871442 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.871377    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zwvhn" podStartSLOduration=4.179808508 podStartE2EDuration="21.871363356s" totalImagesPullingTime="17.691554848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:11.870887917 +0000 UTC m=+23.375365515" watchObservedRunningTime="2026-09-17 20:50:11.871363356 +0000 UTC m=+23.375840986"
Sep 17 20:50:11.888153 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:11.888112    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9dxv5" podStartSLOduration=4.171898824 podStartE2EDuration="21.888098474s" totalImagesPullingTime="17.71619965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:11.887725357 +0000 UTC m=+23.392202952" watchObservedRunningTime="2026-09-17 20:50:11.888098474 +0000 UTC m=+23.392576071"
Sep 17 20:50:12.481442 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:12.481218    2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 20:50:12.617057 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:12.616946    2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T20:50:12.481438824Z","UUID":"0fb00b2b-d6cd-4bca-a168-4ce36d57c832","Handler":null,"Name":"","Endpoint":""}
Sep 17 20:50:12.618680 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:12.618651    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 20:50:12.618808 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:12.618710    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 20:50:12.664009 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:12.663978    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:12.664180 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:12.663989    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:12.664275 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:12.664248    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:50:12.664436 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:12.664410    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:50:12.802001 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:12.801967    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mw75s" event={"ID":"fa88803e-75a6-46c7-b9b4-82e86bbf3540","Type":"ContainerStarted","Data":"a90b848821c22b34099e052a53553abfd7290ef732e11a48db52835965560600"}
Sep 17 20:50:12.805543 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:12.805518    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5" event={"ID":"89ae281b-8074-4ef8-a264-991b56d6e3ff","Type":"ContainerStarted","Data":"347a038db649ce59dc6273e65f6f3a1f48207223a89b2e838d4bae702c560556"}
Sep 17 20:50:12.805680 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:12.805616    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5" event={"ID":"89ae281b-8074-4ef8-a264-991b56d6e3ff","Type":"ContainerStarted","Data":"a7835f50c4c56c80dfc26cb62e1cd48cb4c99294a2ec02b5d730c6bc3928c1fe"}
Sep 17 20:50:12.805680 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:12.805665    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5" event={"ID":"89ae281b-8074-4ef8-a264-991b56d6e3ff","Type":"ContainerStarted","Data":"0a245a486b7824107348eac949f48348f0f8fa514b64427cb2889970458148bd"}
Sep 17 20:50:12.807905 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:12.807867    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb" event={"ID":"1dcc7d98-518b-43c2-85d3-96c7e7af9dde","Type":"ContainerStarted","Data":"639a8d3b12963749ef6364973101f43449f834f6282bb12a66055bcbc4a326c6"}
Sep 17 20:50:13.048676 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:13.048592    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nm7bs"
Sep 17 20:50:13.049484 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:13.049464    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nm7bs"
Sep 17 20:50:13.664281 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:13.664202    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:13.664433 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:13.664314    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:50:13.813447 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:13.813409    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb" event={"ID":"1dcc7d98-518b-43c2-85d3-96c7e7af9dde","Type":"ContainerStarted","Data":"0763d6e14e7c70c9d7e4c28a7865478e1ab0e1ee1060fc85bc09b1549cb28572"}
Sep 17 20:50:13.831730 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:13.831649    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpldb" podStartSLOduration=4.507066965 podStartE2EDuration="23.831633177s" totalImagesPullingTime="19.324566212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:13.831043133 +0000 UTC m=+25.335520729" watchObservedRunningTime="2026-09-17 20:50:13.831633177 +0000 UTC m=+25.336110773"
Sep 17 20:50:14.664206 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:14.664169    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:14.664389 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:14.664169    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:14.664389 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:14.664322    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:50:14.664389 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:14.664361    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:50:14.820876 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:14.820838    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5" event={"ID":"89ae281b-8074-4ef8-a264-991b56d6e3ff","Type":"ContainerStarted","Data":"951b3d897c71ae0d00e543f34245aaa039dd1bca6bb9b96551a633b4a729f399"}
Sep 17 20:50:14.821317 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:14.820851    2571 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 20:50:15.663907 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:15.663868    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:15.664076 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:15.663999    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:50:16.664899 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:16.664676    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:16.665448 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:16.664726    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:16.665448 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:16.664922    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:50:16.665448 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:16.664989    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:50:16.827008 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:16.826974    2571 generic.go:356] "Generic (PLEG): container finished" podID="16b58fb3-0631-4fc1-894a-1d1f7a48b468" containerID="d809ba981c2eb745dab39be68a6fea7371c52bb6faaa60c91e57e0187805db43" exitCode=0
Sep 17 20:50:16.827182 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:16.827054    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47dgq" event={"ID":"16b58fb3-0631-4fc1-894a-1d1f7a48b468","Type":"ContainerDied","Data":"d809ba981c2eb745dab39be68a6fea7371c52bb6faaa60c91e57e0187805db43"}
Sep 17 20:50:16.831128 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:16.831104    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5" event={"ID":"89ae281b-8074-4ef8-a264-991b56d6e3ff","Type":"ContainerStarted","Data":"6761e3c5cf48d830f3e404db614b33c2e56a8de7c3c244e9c71edb4e88cf3980"}
Sep 17 20:50:16.831366 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:16.831353    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:50:16.831526 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:16.831444    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:50:16.831526 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:16.831491    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:50:16.847670 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:16.847644    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:50:16.847800 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:16.847761    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:50:16.847931 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:16.847892    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mw75s" podStartSLOduration=9.159452964 podStartE2EDuration="26.847880023s" totalImagesPullingTime="17.688427059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:13.845953794 +0000 UTC m=+25.350431391" watchObservedRunningTime="2026-09-17 20:50:16.847880023 +0000 UTC m=+28.352357619"
Sep 17 20:50:16.872821 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:16.872769    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5" podStartSLOduration=9.135219024 podStartE2EDuration="26.872755787s" totalImagesPullingTime="17.737536763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:16.871195595 +0000 UTC m=+28.375673193" watchObservedRunningTime="2026-09-17 20:50:16.872755787 +0000 UTC m=+28.377233382"
Sep 17 20:50:17.664443 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:17.664413    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:17.664654 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:17.664542    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:50:18.174635 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:18.174448    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-csv4n"]
Sep 17 20:50:18.175020 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:18.174730    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:18.175020 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:18.174825    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:50:18.177474 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:18.177413    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6742m"]
Sep 17 20:50:18.177599 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:18.177510    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:18.177640 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:18.177622    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:50:18.178186 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:18.178161    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2h2gl"]
Sep 17 20:50:18.178312 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:18.178271    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:18.178378 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:18.178360    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:50:18.837355 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:18.837320    2571 generic.go:356] "Generic (PLEG): container finished" podID="16b58fb3-0631-4fc1-894a-1d1f7a48b468" containerID="999e231adc392a17f787416c31ed60323819512a79a780dd6231e6b0de9aa33b" exitCode=0
Sep 17 20:50:18.837508 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:18.837406    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47dgq" event={"ID":"16b58fb3-0631-4fc1-894a-1d1f7a48b468","Type":"ContainerDied","Data":"999e231adc392a17f787416c31ed60323819512a79a780dd6231e6b0de9aa33b"}
Sep 17 20:50:19.663805 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:19.663777    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:19.664198 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:19.663777    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:19.664198 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:19.663889    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:50:19.664198 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:19.663777    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:19.664198 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:19.663941    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:50:19.664198 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:19.664020    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:50:20.844344 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:20.844312    2571 generic.go:356] "Generic (PLEG): container finished" podID="16b58fb3-0631-4fc1-894a-1d1f7a48b468" containerID="129415ab610da1258eece5ebba5f3afc79167796816da340d6af465bce67072d" exitCode=0
Sep 17 20:50:20.844735 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:20.844381    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47dgq" event={"ID":"16b58fb3-0631-4fc1-894a-1d1f7a48b468","Type":"ContainerDied","Data":"129415ab610da1258eece5ebba5f3afc79167796816da340d6af465bce67072d"}
Sep 17 20:50:21.383746 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:21.383701    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nm7bs"
Sep 17 20:50:21.383897 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:21.383849    2571 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 20:50:21.384375 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:21.384351    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nm7bs"
Sep 17 20:50:21.664572 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:21.664459    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:21.664724 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:21.664607    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:21.664724 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:21.664620    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:21.664724 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:21.664613    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:50:21.664880 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:21.664722    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:50:21.664880 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:21.664803    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:50:23.664347 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:23.664309    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:23.664347 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:23.664342    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:23.664997 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:23.664449    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csv4n" podUID="a47b9a17-f95a-4e22-9584-a457ebbefdaf"
Sep 17 20:50:23.664997 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:23.664573    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6742m" podUID="d2215eda-bca2-49dd-9419-90f9736fdf41"
Sep 17 20:50:23.664997 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:23.664599    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:23.664997 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:23.664701    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2h2gl" podUID="cfe78836-a877-45c6-9b50-c5a481497f71"
Sep 17 20:50:24.303450 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.303154    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs\") pod \"network-metrics-daemon-6742m\" (UID: \"d2215eda-bca2-49dd-9419-90f9736fdf41\") " pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:24.303450 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.303432    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret\") pod \"global-pull-secret-syncer-2h2gl\" (UID: \"cfe78836-a877-45c6-9b50-c5a481497f71\") " pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:24.303707 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:24.303316    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:50:24.303707 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:24.303583    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:50:24.303707 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:24.303585    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs podName:d2215eda-bca2-49dd-9419-90f9736fdf41 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:56.303543958 +0000 UTC m=+67.808021548 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs") pod "network-metrics-daemon-6742m" (UID: "d2215eda-bca2-49dd-9419-90f9736fdf41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:50:24.303707 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:24.303641    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret podName:cfe78836-a877-45c6-9b50-c5a481497f71 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:56.303624297 +0000 UTC m=+67.808101887 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret") pod "global-pull-secret-syncer-2h2gl" (UID: "cfe78836-a877-45c6-9b50-c5a481497f71") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:50:24.404213 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.404181    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9gk\" (UniqueName: \"kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk\") pod \"network-check-target-csv4n\" (UID: \"a47b9a17-f95a-4e22-9584-a457ebbefdaf\") " pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:24.404378 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:24.404313    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:50:24.404378 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:24.404330    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:50:24.404378 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:24.404342    2571 projected.go:196] Error preparing data for projected volume kube-api-access-7m9gk for pod openshift-network-diagnostics/network-check-target-csv4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:50:24.404510 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:24.404403    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk podName:a47b9a17-f95a-4e22-9584-a457ebbefdaf nodeName:}" failed. No retries permitted until 2026-09-17 20:50:56.404387097 +0000 UTC m=+67.908864672 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7m9gk" (UniqueName: "kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk") pod "network-check-target-csv4n" (UID: "a47b9a17-f95a-4e22-9584-a457ebbefdaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:50:24.427678 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.427647    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-94.ec2.internal" event="NodeReady"
Sep 17 20:50:24.427831 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.427800    2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 20:50:24.479181 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.479150    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mp5gn"]
Sep 17 20:50:24.509768 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.509734    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-f2bdn"]
Sep 17 20:50:24.509935 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.509872    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:24.513033 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.513007    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 20:50:24.513229 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.513203    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k4498\""
Sep 17 20:50:24.513342 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.513236    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 20:50:24.525827 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.525803    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mp5gn"]
Sep 17 20:50:24.525935 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.525857    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f2bdn"
Sep 17 20:50:24.525935 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.525871    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f2bdn"]
Sep 17 20:50:24.528881 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.528854    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 20:50:24.529353 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.529326    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 20:50:24.529522 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.529497    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wc77b\""
Sep 17 20:50:24.530982 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.530957    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 20:50:24.605817 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.605784    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/325ad1ea-7622-490d-82c3-ff73e8ff5dda-config-volume\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:24.605817 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.605820    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07aff763-5b60-4dda-89de-f72e2896a715-cert\") pod \"ingress-canary-f2bdn\" (UID: \"07aff763-5b60-4dda-89de-f72e2896a715\") " pod="openshift-ingress-canary/ingress-canary-f2bdn"
Sep 17 20:50:24.606060 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.605845    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:24.606060 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.605928    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcf8f\" (UniqueName: \"kubernetes.io/projected/325ad1ea-7622-490d-82c3-ff73e8ff5dda-kube-api-access-rcf8f\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:24.606060 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.605996    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrqkh\" (UniqueName: \"kubernetes.io/projected/07aff763-5b60-4dda-89de-f72e2896a715-kube-api-access-qrqkh\") pod \"ingress-canary-f2bdn\" (UID: \"07aff763-5b60-4dda-89de-f72e2896a715\") " pod="openshift-ingress-canary/ingress-canary-f2bdn"
Sep 17 20:50:24.606060 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.606032    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/325ad1ea-7622-490d-82c3-ff73e8ff5dda-tmp-dir\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:24.706483 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.706433    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/325ad1ea-7622-490d-82c3-ff73e8ff5dda-tmp-dir\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:24.706905 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.706498    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/325ad1ea-7622-490d-82c3-ff73e8ff5dda-config-volume\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:24.706905 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.706525    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07aff763-5b60-4dda-89de-f72e2896a715-cert\") pod \"ingress-canary-f2bdn\" (UID: \"07aff763-5b60-4dda-89de-f72e2896a715\") " pod="openshift-ingress-canary/ingress-canary-f2bdn"
Sep 17 20:50:24.706905 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.706567    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:24.706905 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.706618    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcf8f\" (UniqueName: \"kubernetes.io/projected/325ad1ea-7622-490d-82c3-ff73e8ff5dda-kube-api-access-rcf8f\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:24.706905 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.706676    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrqkh\" (UniqueName: \"kubernetes.io/projected/07aff763-5b60-4dda-89de-f72e2896a715-kube-api-access-qrqkh\") pod \"ingress-canary-f2bdn\" (UID: \"07aff763-5b60-4dda-89de-f72e2896a715\") " pod="openshift-ingress-canary/ingress-canary-f2bdn"
Sep 17 20:50:24.706905 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:24.706752    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:24.706905 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:24.706818    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls podName:325ad1ea-7622-490d-82c3-ff73e8ff5dda nodeName:}" failed. No retries permitted until 2026-09-17 20:50:25.206799394 +0000 UTC m=+36.711276981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls") pod "dns-default-mp5gn" (UID: "325ad1ea-7622-490d-82c3-ff73e8ff5dda") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:24.707127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.707102    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/325ad1ea-7622-490d-82c3-ff73e8ff5dda-tmp-dir\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:24.707286 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.707266    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/325ad1ea-7622-490d-82c3-ff73e8ff5dda-config-volume\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:24.711820 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.711779    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07aff763-5b60-4dda-89de-f72e2896a715-cert\") pod \"ingress-canary-f2bdn\" (UID: \"07aff763-5b60-4dda-89de-f72e2896a715\") " pod="openshift-ingress-canary/ingress-canary-f2bdn"
Sep 17 20:50:24.717663 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.717638    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrqkh\" (UniqueName: \"kubernetes.io/projected/07aff763-5b60-4dda-89de-f72e2896a715-kube-api-access-qrqkh\") pod \"ingress-canary-f2bdn\" (UID: \"07aff763-5b60-4dda-89de-f72e2896a715\") " pod="openshift-ingress-canary/ingress-canary-f2bdn"
Sep 17 20:50:24.717975 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.717950    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcf8f\" (UniqueName: \"kubernetes.io/projected/325ad1ea-7622-490d-82c3-ff73e8ff5dda-kube-api-access-rcf8f\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:24.837123 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:24.837086    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f2bdn"
Sep 17 20:50:25.156005 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:25.155972    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f2bdn"]
Sep 17 20:50:25.160306 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:50:25.160283    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07aff763_5b60_4dda_89de_f72e2896a715.slice/crio-7334c53b3e43b8501e87ee4011f638454eac6e79fb5d7b545037eebe5c3617b1 WatchSource:0}: Error finding container 7334c53b3e43b8501e87ee4011f638454eac6e79fb5d7b545037eebe5c3617b1: Status 404 returned error can't find the container with id 7334c53b3e43b8501e87ee4011f638454eac6e79fb5d7b545037eebe5c3617b1
Sep 17 20:50:25.210970 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:25.210945    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:25.211107 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:25.211093    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:25.211157 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:25.211152    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls podName:325ad1ea-7622-490d-82c3-ff73e8ff5dda nodeName:}" failed. No retries permitted until 2026-09-17 20:50:26.211132939 +0000 UTC m=+37.715610532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls") pod "dns-default-mp5gn" (UID: "325ad1ea-7622-490d-82c3-ff73e8ff5dda") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:25.664179 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:25.664148    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:25.664376 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:25.664152    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:25.664376 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:25.664301    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:25.668436 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:25.668410    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 20:50:25.668698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:25.668678    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9vrtf\""
Sep 17 20:50:25.668844 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:25.668755    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 20:50:25.668844 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:25.668806    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 20:50:25.668971 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:25.668867    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kfcvc\""
Sep 17 20:50:25.668971 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:25.668756    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 20:50:25.858425 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:25.858385    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f2bdn" event={"ID":"07aff763-5b60-4dda-89de-f72e2896a715","Type":"ContainerStarted","Data":"7334c53b3e43b8501e87ee4011f638454eac6e79fb5d7b545037eebe5c3617b1"}
Sep 17 20:50:25.861852 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:25.861825    2571 generic.go:356] "Generic (PLEG): container finished" podID="16b58fb3-0631-4fc1-894a-1d1f7a48b468" containerID="f53b31ef6b02d8901fb6cfa2b28602c2494097bd3eadc4cf1886c73a5d469ac7" exitCode=0
Sep 17 20:50:25.861975 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:25.861890    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47dgq" event={"ID":"16b58fb3-0631-4fc1-894a-1d1f7a48b468","Type":"ContainerDied","Data":"f53b31ef6b02d8901fb6cfa2b28602c2494097bd3eadc4cf1886c73a5d469ac7"}
Sep 17 20:50:26.220045 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:26.219967    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:26.220202 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:26.220115    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:26.220251 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:26.220204    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls podName:325ad1ea-7622-490d-82c3-ff73e8ff5dda nodeName:}" failed. No retries permitted until 2026-09-17 20:50:28.220181119 +0000 UTC m=+39.724658717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls") pod "dns-default-mp5gn" (UID: "325ad1ea-7622-490d-82c3-ff73e8ff5dda") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:26.868402 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:26.868197    2571 generic.go:356] "Generic (PLEG): container finished" podID="16b58fb3-0631-4fc1-894a-1d1f7a48b468" containerID="07fb290458d3840046923cff3173532264621506ed721088e431204aab34b119" exitCode=0
Sep 17 20:50:26.868837 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:26.868278    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47dgq" event={"ID":"16b58fb3-0631-4fc1-894a-1d1f7a48b468","Type":"ContainerDied","Data":"07fb290458d3840046923cff3173532264621506ed721088e431204aab34b119"}
Sep 17 20:50:27.872484 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:27.872444    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f2bdn" event={"ID":"07aff763-5b60-4dda-89de-f72e2896a715","Type":"ContainerStarted","Data":"d5acd90ca3f9493bb89ec66d7d29604a037f4cec996ad79e692fed2f32e54da7"}
Sep 17 20:50:27.876109 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:27.876082    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-47dgq" event={"ID":"16b58fb3-0631-4fc1-894a-1d1f7a48b468","Type":"ContainerStarted","Data":"4c853ca109e7837219c95536c0c154231d321e1b1bcfc6204330cbfdd2c2ba07"}
Sep 17 20:50:27.889666 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:27.889622    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-f2bdn" podStartSLOduration=1.360521155 podStartE2EDuration="3.889608081s" totalImagesPullingTime="2.529086926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:27.889172524 +0000 UTC m=+39.393650119" watchObservedRunningTime="2026-09-17 20:50:27.889608081 +0000 UTC m=+39.394085677"
Sep 17 20:50:27.911219 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:27.911170    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-47dgq" podStartSLOduration=9.345959529 podStartE2EDuration="37.911154024s" totalImagesPullingTime="28.565194495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:27.910907153 +0000 UTC m=+39.415384749" watchObservedRunningTime="2026-09-17 20:50:27.911154024 +0000 UTC m=+39.415631619"
Sep 17 20:50:28.234005 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:28.233964    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:28.234210 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:28.234130    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:28.234283 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:28.234218    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls podName:325ad1ea-7622-490d-82c3-ff73e8ff5dda nodeName:}" failed. No retries permitted until 2026-09-17 20:50:32.234188576 +0000 UTC m=+43.738666167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls") pod "dns-default-mp5gn" (UID: "325ad1ea-7622-490d-82c3-ff73e8ff5dda") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:32.261112 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:32.261063    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:32.261782 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:32.261241    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:32.261782 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:32.261331    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls podName:325ad1ea-7622-490d-82c3-ff73e8ff5dda nodeName:}" failed. No retries permitted until 2026-09-17 20:50:40.261308757 +0000 UTC m=+51.765786355 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls") pod "dns-default-mp5gn" (UID: "325ad1ea-7622-490d-82c3-ff73e8ff5dda") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:40.317523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:40.317482    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:40.318173 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:40.318146    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:40.318301 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:40.318287    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls podName:325ad1ea-7622-490d-82c3-ff73e8ff5dda nodeName:}" failed. No retries permitted until 2026-09-17 20:50:56.318262485 +0000 UTC m=+67.822740080 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls") pod "dns-default-mp5gn" (UID: "325ad1ea-7622-490d-82c3-ff73e8ff5dda") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:48.859332 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:48.859303    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wlt5"
Sep 17 20:50:56.326603 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.326528    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret\") pod \"global-pull-secret-syncer-2h2gl\" (UID: \"cfe78836-a877-45c6-9b50-c5a481497f71\") " pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:56.326999 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.326630    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:50:56.326999 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.326653    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs\") pod \"network-metrics-daemon-6742m\" (UID: \"d2215eda-bca2-49dd-9419-90f9736fdf41\") " pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:50:56.326999 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:56.326768    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:56.326999 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:56.326848    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls podName:325ad1ea-7622-490d-82c3-ff73e8ff5dda nodeName:}" failed. No retries permitted until 2026-09-17 20:51:28.326827504 +0000 UTC m=+99.831305094 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls") pod "dns-default-mp5gn" (UID: "325ad1ea-7622-490d-82c3-ff73e8ff5dda") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:56.329603 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.329580    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 20:50:56.329603 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.329599    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 20:50:56.337043 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:56.337028    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 20:50:56.337128 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:50:56.337090    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs podName:d2215eda-bca2-49dd-9419-90f9736fdf41 nodeName:}" failed. No retries permitted until 2026-09-17 20:52:00.337075167 +0000 UTC m=+131.841552741 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs") pod "network-metrics-daemon-6742m" (UID: "d2215eda-bca2-49dd-9419-90f9736fdf41") : secret "metrics-daemon-secret" not found
Sep 17 20:50:56.340312 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.340296    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cfe78836-a877-45c6-9b50-c5a481497f71-original-pull-secret\") pod \"global-pull-secret-syncer-2h2gl\" (UID: \"cfe78836-a877-45c6-9b50-c5a481497f71\") " pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:56.427670 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.427630    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9gk\" (UniqueName: \"kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk\") pod \"network-check-target-csv4n\" (UID: \"a47b9a17-f95a-4e22-9584-a457ebbefdaf\") " pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:56.431779 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.431760    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 20:50:56.441010 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.440989    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 20:50:56.451965 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.451936    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m9gk\" (UniqueName: \"kubernetes.io/projected/a47b9a17-f95a-4e22-9584-a457ebbefdaf-kube-api-access-7m9gk\") pod \"network-check-target-csv4n\" (UID: \"a47b9a17-f95a-4e22-9584-a457ebbefdaf\") " pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:56.589803 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.589721    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kfcvc\""
Sep 17 20:50:56.591671 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.591654    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2h2gl"
Sep 17 20:50:56.597583 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.597559    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:50:56.865923 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.865852    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2h2gl"]
Sep 17 20:50:56.868442 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.868417    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-csv4n"]
Sep 17 20:50:56.869732 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:50:56.869699    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe78836_a877_45c6_9b50_c5a481497f71.slice/crio-d07f2a1263e3b3d4c05d72e7d9dc0679eee610afcce232e6337a3f827da80dc1 WatchSource:0}: Error finding container d07f2a1263e3b3d4c05d72e7d9dc0679eee610afcce232e6337a3f827da80dc1: Status 404 returned error can't find the container with id d07f2a1263e3b3d4c05d72e7d9dc0679eee610afcce232e6337a3f827da80dc1
Sep 17 20:50:56.870884 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:50:56.870858    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47b9a17_f95a_4e22_9584_a457ebbefdaf.slice/crio-17dd9a9644e4c97a5923c3f82f6f6a2417fca9aa6816973902c539af663b30e0 WatchSource:0}: Error finding container 17dd9a9644e4c97a5923c3f82f6f6a2417fca9aa6816973902c539af663b30e0: Status 404 returned error can't find the container with id 17dd9a9644e4c97a5923c3f82f6f6a2417fca9aa6816973902c539af663b30e0
Sep 17 20:50:56.944708 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.944669    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2h2gl" event={"ID":"cfe78836-a877-45c6-9b50-c5a481497f71","Type":"ContainerStarted","Data":"d07f2a1263e3b3d4c05d72e7d9dc0679eee610afcce232e6337a3f827da80dc1"}
Sep 17 20:50:56.945867 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:50:56.945845    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-csv4n" event={"ID":"a47b9a17-f95a-4e22-9584-a457ebbefdaf","Type":"ContainerStarted","Data":"17dd9a9644e4c97a5923c3f82f6f6a2417fca9aa6816973902c539af663b30e0"}
Sep 17 20:51:01.296728 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.296599    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5"]
Sep 17 20:51:01.298809 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.298788    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5"
Sep 17 20:51:01.301698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.301537    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 20:51:01.301698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.301536    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:51:01.301698 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.301588    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 20:51:01.302503 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.302485    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 20:51:01.302626 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.302512    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wmmqs\""
Sep 17 20:51:01.310268 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.310246    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss"]
Sep 17 20:51:01.312210 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.312183    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss"
Sep 17 20:51:01.312792 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.312769    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-4rvqw"]
Sep 17 20:51:01.315178 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.315156    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5"]
Sep 17 20:51:01.315280 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.315266    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.318104 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.318064    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 20:51:01.319129 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.319112    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 20:51:01.321705 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.321688    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:51:01.322266 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.322250    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vh776\""
Sep 17 20:51:01.323911 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.323891    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xldv6\""
Sep 17 20:51:01.324203 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.324186    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 20:51:01.324297 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.324212    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 20:51:01.324450 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.324433    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 20:51:01.324570 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.324489    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 20:51:01.353948 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.353909    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss"]
Sep 17 20:51:01.354173 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.354153    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 20:51:01.358205 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.358182    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-4rvqw"]
Sep 17 20:51:01.367074 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.367047    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad635b09-43a6-4d29-b7c0-792f8327751c-serving-cert\") pod \"service-ca-operator-57cdc4b768-fczn5\" (UID: \"ad635b09-43a6-4d29-b7c0-792f8327751c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5"
Sep 17 20:51:01.367186 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.367085    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bf9e322-db1c-4373-986b-d4d95d2dad3c-tmp\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.367186 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.367114    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad635b09-43a6-4d29-b7c0-792f8327751c-config\") pod \"service-ca-operator-57cdc4b768-fczn5\" (UID: \"ad635b09-43a6-4d29-b7c0-792f8327751c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5"
Sep 17 20:51:01.367186 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.367140    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j26h\" (UniqueName: \"kubernetes.io/projected/964e7579-62e6-4dd0-92a1-05569d75d125-kube-api-access-6j26h\") pod \"cluster-samples-operator-685d7d68b5-kcmss\" (UID: \"964e7579-62e6-4dd0-92a1-05569d75d125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss"
Sep 17 20:51:01.367327 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.367241    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8bf9e322-db1c-4373-986b-d4d95d2dad3c-snapshots\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.367327 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.367283    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bf9e322-db1c-4373-986b-d4d95d2dad3c-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.367327 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.367310    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-kcmss\" (UID: \"964e7579-62e6-4dd0-92a1-05569d75d125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss"
Sep 17 20:51:01.367491 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.367336    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bf9e322-db1c-4373-986b-d4d95d2dad3c-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.367491 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.367379    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbc9n\" (UniqueName: \"kubernetes.io/projected/ad635b09-43a6-4d29-b7c0-792f8327751c-kube-api-access-lbc9n\") pod \"service-ca-operator-57cdc4b768-fczn5\" (UID: \"ad635b09-43a6-4d29-b7c0-792f8327751c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5"
Sep 17 20:51:01.367491 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.367398    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8bf9e322-db1c-4373-986b-d4d95d2dad3c-serving-cert\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.367491 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.367422    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdzfn\" (UniqueName: \"kubernetes.io/projected/8bf9e322-db1c-4373-986b-d4d95d2dad3c-kube-api-access-sdzfn\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.414603 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.414563    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5"]
Sep 17 20:51:01.416523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.416501    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5"
Sep 17 20:51:01.416977 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.416955    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"]
Sep 17 20:51:01.418596 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.418577    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:01.419216 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.419188    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 20:51:01.419301 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.419260    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6mhwd\""
Sep 17 20:51:01.419562 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.419527    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 20:51:01.419704 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.419690    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:51:01.419811 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.419794    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 20:51:01.421375 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.421359    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 20:51:01.421916 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.421898    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 20:51:01.422417 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.422399    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 20:51:01.424255 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.424235    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 20:51:01.424459 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.424446    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gc4hg\""
Sep 17 20:51:01.431848 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.431709    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5"]
Sep 17 20:51:01.442780 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.442748    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"]
Sep 17 20:51:01.468205 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468168    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8bf9e322-db1c-4373-986b-d4d95d2dad3c-snapshots\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.468205 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468209    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bf9e322-db1c-4373-986b-d4d95d2dad3c-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.468205 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468227    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-kcmss\" (UID: \"964e7579-62e6-4dd0-92a1-05569d75d125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss"
Sep 17 20:51:01.468469 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468248    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rgxgj\" (UID: \"5696194e-5b60-4427-9147-1c42eb17a7ed\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:01.468469 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468272    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9jzc\" (UniqueName: \"kubernetes.io/projected/5696194e-5b60-4427-9147-1c42eb17a7ed-kube-api-access-v9jzc\") pod \"cluster-monitoring-operator-847b8bd8d7-rgxgj\" (UID: \"5696194e-5b60-4427-9147-1c42eb17a7ed\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:01.468469 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468300    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bf9e322-db1c-4373-986b-d4d95d2dad3c-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.468469 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:01.468354    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 20:51:01.468469 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468384    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbc9n\" (UniqueName: \"kubernetes.io/projected/ad635b09-43a6-4d29-b7c0-792f8327751c-kube-api-access-lbc9n\") pod \"service-ca-operator-57cdc4b768-fczn5\" (UID: \"ad635b09-43a6-4d29-b7c0-792f8327751c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5"
Sep 17 20:51:01.468469 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:01.468422    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls podName:964e7579-62e6-4dd0-92a1-05569d75d125 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:01.968402681 +0000 UTC m=+73.472880255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-kcmss" (UID: "964e7579-62e6-4dd0-92a1-05569d75d125") : secret "samples-operator-tls" not found
Sep 17 20:51:01.468469 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468451    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8bf9e322-db1c-4373-986b-d4d95d2dad3c-serving-cert\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.468858 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468476    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdzfn\" (UniqueName: \"kubernetes.io/projected/8bf9e322-db1c-4373-986b-d4d95d2dad3c-kube-api-access-sdzfn\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.468858 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468505    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5696194e-5b60-4427-9147-1c42eb17a7ed-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-rgxgj\" (UID: \"5696194e-5b60-4427-9147-1c42eb17a7ed\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:01.468858 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468567    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad635b09-43a6-4d29-b7c0-792f8327751c-serving-cert\") pod \"service-ca-operator-57cdc4b768-fczn5\" (UID: \"ad635b09-43a6-4d29-b7c0-792f8327751c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5"
Sep 17 20:51:01.468858 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468595    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/374b4cca-e914-4b51-a833-32dcbcd655f2-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-xbdx5\" (UID: \"374b4cca-e914-4b51-a833-32dcbcd655f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5"
Sep 17 20:51:01.468858 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468626    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/374b4cca-e914-4b51-a833-32dcbcd655f2-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-xbdx5\" (UID: \"374b4cca-e914-4b51-a833-32dcbcd655f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5"
Sep 17 20:51:01.468858 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468655    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bf9e322-db1c-4373-986b-d4d95d2dad3c-tmp\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.468858 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468697    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad635b09-43a6-4d29-b7c0-792f8327751c-config\") pod \"service-ca-operator-57cdc4b768-fczn5\" (UID: \"ad635b09-43a6-4d29-b7c0-792f8327751c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5"
Sep 17 20:51:01.468858 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468729    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j26h\" (UniqueName: \"kubernetes.io/projected/964e7579-62e6-4dd0-92a1-05569d75d125-kube-api-access-6j26h\") pod \"cluster-samples-operator-685d7d68b5-kcmss\" (UID: \"964e7579-62e6-4dd0-92a1-05569d75d125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss"
Sep 17 20:51:01.468858 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468757    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfn4x\" (UniqueName: \"kubernetes.io/projected/374b4cca-e914-4b51-a833-32dcbcd655f2-kube-api-access-mfn4x\") pod \"kube-storage-version-migrator-operator-55889f7b47-xbdx5\" (UID: \"374b4cca-e914-4b51-a833-32dcbcd655f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5"
Sep 17 20:51:01.469301 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468877    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8bf9e322-db1c-4373-986b-d4d95d2dad3c-snapshots\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.469301 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468885    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8bf9e322-db1c-4373-986b-d4d95d2dad3c-tmp\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.469301 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.468975    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bf9e322-db1c-4373-986b-d4d95d2dad3c-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.469453 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.469306    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bf9e322-db1c-4373-986b-d4d95d2dad3c-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.469642 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.469618    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad635b09-43a6-4d29-b7c0-792f8327751c-config\") pod \"service-ca-operator-57cdc4b768-fczn5\" (UID: \"ad635b09-43a6-4d29-b7c0-792f8327751c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5"
Sep 17 20:51:01.471296 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.471278    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8bf9e322-db1c-4373-986b-d4d95d2dad3c-serving-cert\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.471647 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.471627    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad635b09-43a6-4d29-b7c0-792f8327751c-serving-cert\") pod \"service-ca-operator-57cdc4b768-fczn5\" (UID: \"ad635b09-43a6-4d29-b7c0-792f8327751c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5"
Sep 17 20:51:01.477753 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.477730    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbc9n\" (UniqueName: \"kubernetes.io/projected/ad635b09-43a6-4d29-b7c0-792f8327751c-kube-api-access-lbc9n\") pod \"service-ca-operator-57cdc4b768-fczn5\" (UID: \"ad635b09-43a6-4d29-b7c0-792f8327751c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5"
Sep 17 20:51:01.479717 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.479698    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdzfn\" (UniqueName: \"kubernetes.io/projected/8bf9e322-db1c-4373-986b-d4d95d2dad3c-kube-api-access-sdzfn\") pod \"insights-operator-6f9c644dd9-4rvqw\" (UID: \"8bf9e322-db1c-4373-986b-d4d95d2dad3c\") " pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.479804 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.479777    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j26h\" (UniqueName: \"kubernetes.io/projected/964e7579-62e6-4dd0-92a1-05569d75d125-kube-api-access-6j26h\") pod \"cluster-samples-operator-685d7d68b5-kcmss\" (UID: \"964e7579-62e6-4dd0-92a1-05569d75d125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss"
Sep 17 20:51:01.569032 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.569000    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5696194e-5b60-4427-9147-1c42eb17a7ed-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-rgxgj\" (UID: \"5696194e-5b60-4427-9147-1c42eb17a7ed\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:01.569032 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.569036    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/374b4cca-e914-4b51-a833-32dcbcd655f2-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-xbdx5\" (UID: \"374b4cca-e914-4b51-a833-32dcbcd655f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5"
Sep 17 20:51:01.569169 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.569054    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/374b4cca-e914-4b51-a833-32dcbcd655f2-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-xbdx5\" (UID: \"374b4cca-e914-4b51-a833-32dcbcd655f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5"
Sep 17 20:51:01.569169 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.569076    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfn4x\" (UniqueName: \"kubernetes.io/projected/374b4cca-e914-4b51-a833-32dcbcd655f2-kube-api-access-mfn4x\") pod \"kube-storage-version-migrator-operator-55889f7b47-xbdx5\" (UID: \"374b4cca-e914-4b51-a833-32dcbcd655f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5"
Sep 17 20:51:01.569169 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.569119    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rgxgj\" (UID: \"5696194e-5b60-4427-9147-1c42eb17a7ed\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:01.569169 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.569135    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9jzc\" (UniqueName: \"kubernetes.io/projected/5696194e-5b60-4427-9147-1c42eb17a7ed-kube-api-access-v9jzc\") pod \"cluster-monitoring-operator-847b8bd8d7-rgxgj\" (UID: \"5696194e-5b60-4427-9147-1c42eb17a7ed\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:01.569289 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:01.569248    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:01.569321 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:01.569303    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls podName:5696194e-5b60-4427-9147-1c42eb17a7ed nodeName:}" failed. No retries permitted until 2026-09-17 20:51:02.069285851 +0000 UTC m=+73.573763426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-rgxgj" (UID: "5696194e-5b60-4427-9147-1c42eb17a7ed") : secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:01.569719 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.569692    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/374b4cca-e914-4b51-a833-32dcbcd655f2-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-xbdx5\" (UID: \"374b4cca-e914-4b51-a833-32dcbcd655f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5"
Sep 17 20:51:01.569813 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.569795    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5696194e-5b60-4427-9147-1c42eb17a7ed-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-rgxgj\" (UID: \"5696194e-5b60-4427-9147-1c42eb17a7ed\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:01.571596 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.571574    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/374b4cca-e914-4b51-a833-32dcbcd655f2-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-xbdx5\" (UID: \"374b4cca-e914-4b51-a833-32dcbcd655f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5"
Sep 17 20:51:01.577095 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.577072    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9jzc\" (UniqueName: \"kubernetes.io/projected/5696194e-5b60-4427-9147-1c42eb17a7ed-kube-api-access-v9jzc\") pod \"cluster-monitoring-operator-847b8bd8d7-rgxgj\" (UID: \"5696194e-5b60-4427-9147-1c42eb17a7ed\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:01.578517 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.578496    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfn4x\" (UniqueName: \"kubernetes.io/projected/374b4cca-e914-4b51-a833-32dcbcd655f2-kube-api-access-mfn4x\") pod \"kube-storage-version-migrator-operator-55889f7b47-xbdx5\" (UID: \"374b4cca-e914-4b51-a833-32dcbcd655f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5"
Sep 17 20:51:01.609982 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.609951    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5"
Sep 17 20:51:01.629537 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.629503    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw"
Sep 17 20:51:01.729009 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.728976    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5"
Sep 17 20:51:01.886276 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.886244    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5"]
Sep 17 20:51:01.889332 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:01.889297    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad635b09_43a6_4d29_b7c0_792f8327751c.slice/crio-4a1484592e888eda735ba53f6758d4aa9806b660cad9df80af2767ee20ed9d87 WatchSource:0}: Error finding container 4a1484592e888eda735ba53f6758d4aa9806b660cad9df80af2767ee20ed9d87: Status 404 returned error can't find the container with id 4a1484592e888eda735ba53f6758d4aa9806b660cad9df80af2767ee20ed9d87
Sep 17 20:51:01.910102 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.910078    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-4rvqw"]
Sep 17 20:51:01.912752 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:01.912728    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf9e322_db1c_4373_986b_d4d95d2dad3c.slice/crio-8a714f83f4ac67ea1fcde053c78f9555866fd03010207bf06e2b2b1ddebdf231 WatchSource:0}: Error finding container 8a714f83f4ac67ea1fcde053c78f9555866fd03010207bf06e2b2b1ddebdf231: Status 404 returned error can't find the container with id 8a714f83f4ac67ea1fcde053c78f9555866fd03010207bf06e2b2b1ddebdf231
Sep 17 20:51:01.960914 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.960879    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5" event={"ID":"ad635b09-43a6-4d29-b7c0-792f8327751c","Type":"ContainerStarted","Data":"4a1484592e888eda735ba53f6758d4aa9806b660cad9df80af2767ee20ed9d87"}
Sep 17 20:51:01.962333 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.962309    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-csv4n" event={"ID":"a47b9a17-f95a-4e22-9584-a457ebbefdaf","Type":"ContainerStarted","Data":"0dc4746c48cfc45a1a47c8843469648b4c4c42c24881a5e4138e799b4bf77bf3"}
Sep 17 20:51:01.962440 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.962391    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:51:01.963999 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.963977    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2h2gl" event={"ID":"cfe78836-a877-45c6-9b50-c5a481497f71","Type":"ContainerStarted","Data":"3d96d1f2274205c38ab339cd90049c4fc166faf82d0e2a90418649da50adc083"}
Sep 17 20:51:01.965074 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.965053    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw" event={"ID":"8bf9e322-db1c-4373-986b-d4d95d2dad3c","Type":"ContainerStarted","Data":"8a714f83f4ac67ea1fcde053c78f9555866fd03010207bf06e2b2b1ddebdf231"}
Sep 17 20:51:01.972384 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.972363    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-kcmss\" (UID: \"964e7579-62e6-4dd0-92a1-05569d75d125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss"
Sep 17 20:51:01.972487 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:01.972474    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 20:51:01.972572 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:01.972532    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls podName:964e7579-62e6-4dd0-92a1-05569d75d125 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:02.972513879 +0000 UTC m=+74.476991473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-kcmss" (UID: "964e7579-62e6-4dd0-92a1-05569d75d125") : secret "samples-operator-tls" not found
Sep 17 20:51:01.979716 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.979677    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-csv4n" podStartSLOduration=67.375855121 podStartE2EDuration="1m11.979663578s" totalImagesPullingTime="4.603808457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:01.978146435 +0000 UTC m=+73.482624032" watchObservedRunningTime="2026-09-17 20:51:01.979663578 +0000 UTC m=+73.484141174"
Sep 17 20:51:01.994104 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.994075    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5"]
Sep 17 20:51:01.996462 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:01.996418    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2h2gl" podStartSLOduration=67.387222739 podStartE2EDuration="1m11.996401932s" totalImagesPullingTime="4.609179193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:01.995443097 +0000 UTC m=+73.499920692" watchObservedRunningTime="2026-09-17 20:51:01.996401932 +0000 UTC m=+73.500879519"
Sep 17 20:51:01.996988 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:01.996960    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374b4cca_e914_4b51_a833_32dcbcd655f2.slice/crio-1ced7178b821120de8d649636a1178686b08696c86f470d41ddb82065375b01d WatchSource:0}: Error finding container 1ced7178b821120de8d649636a1178686b08696c86f470d41ddb82065375b01d: Status 404 returned error can't find the container with id 1ced7178b821120de8d649636a1178686b08696c86f470d41ddb82065375b01d
Sep 17 20:51:02.073654 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:02.073620    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rgxgj\" (UID: \"5696194e-5b60-4427-9147-1c42eb17a7ed\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:02.073820 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:02.073717    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:02.073820 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:02.073770    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls podName:5696194e-5b60-4427-9147-1c42eb17a7ed nodeName:}" failed. No retries permitted until 2026-09-17 20:51:03.073756933 +0000 UTC m=+74.578234520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-rgxgj" (UID: "5696194e-5b60-4427-9147-1c42eb17a7ed") : secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:02.970742 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:02.970248    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5" event={"ID":"374b4cca-e914-4b51-a833-32dcbcd655f2","Type":"ContainerStarted","Data":"1ced7178b821120de8d649636a1178686b08696c86f470d41ddb82065375b01d"}
Sep 17 20:51:02.982537 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:02.982495    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-kcmss\" (UID: \"964e7579-62e6-4dd0-92a1-05569d75d125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss"
Sep 17 20:51:02.983100 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:02.983083    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 20:51:02.983184 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:02.983144    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls podName:964e7579-62e6-4dd0-92a1-05569d75d125 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:04.983126707 +0000 UTC m=+76.487604284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-kcmss" (UID: "964e7579-62e6-4dd0-92a1-05569d75d125") : secret "samples-operator-tls" not found
Sep 17 20:51:03.083931 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:03.083889    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rgxgj\" (UID: \"5696194e-5b60-4427-9147-1c42eb17a7ed\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:03.084100 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:03.084062    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:03.084181 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:03.084127    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls podName:5696194e-5b60-4427-9147-1c42eb17a7ed nodeName:}" failed. No retries permitted until 2026-09-17 20:51:05.084107678 +0000 UTC m=+76.588585265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-rgxgj" (UID: "5696194e-5b60-4427-9147-1c42eb17a7ed") : secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:04.978334 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:04.978301    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw" event={"ID":"8bf9e322-db1c-4373-986b-d4d95d2dad3c","Type":"ContainerStarted","Data":"9bb01eee99ff051874df56d20a52fd7faa46834f37f8507d3968ad2ae3684f8e"}
Sep 17 20:51:04.980034 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:04.980004    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5" event={"ID":"ad635b09-43a6-4d29-b7c0-792f8327751c","Type":"ContainerStarted","Data":"d533bed4a374c9af9f04c758184bc693b8f1df2640ff3f4a04ad582ede38f45f"}
Sep 17 20:51:04.981784 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:04.981757    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5" event={"ID":"374b4cca-e914-4b51-a833-32dcbcd655f2","Type":"ContainerStarted","Data":"565fbfe56f92e2060115243f90330feda07569947b27d1424e9a64a22adba167"}
Sep 17 20:51:05.000580 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.000501    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw" podStartSLOduration=1.049950115 podStartE2EDuration="4.000480871s" totalImagesPullingTime="2.950530756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:04.998808923 +0000 UTC m=+76.503286522" watchObservedRunningTime="2026-09-17 20:51:05.000480871 +0000 UTC m=+76.504958472"
Sep 17 20:51:05.002229 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.001065    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-kcmss\" (UID: \"964e7579-62e6-4dd0-92a1-05569d75d125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss"
Sep 17 20:51:05.002229 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:05.001182    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 20:51:05.002229 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:05.001248    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls podName:964e7579-62e6-4dd0-92a1-05569d75d125 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:09.001231884 +0000 UTC m=+80.505709465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-kcmss" (UID: "964e7579-62e6-4dd0-92a1-05569d75d125") : secret "samples-operator-tls" not found
Sep 17 20:51:05.016094 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.016048    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5" podStartSLOduration=1.043921881 podStartE2EDuration="4.016032403s" totalImagesPullingTime="2.972110522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:05.01516684 +0000 UTC m=+76.519644437" watchObservedRunningTime="2026-09-17 20:51:05.016032403 +0000 UTC m=+76.520509991"
Sep 17 20:51:05.036457 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.036323    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5" podStartSLOduration=1.176070908 podStartE2EDuration="4.036299099s" totalImagesPullingTime="2.860228191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:05.035322701 +0000 UTC m=+76.539800300" watchObservedRunningTime="2026-09-17 20:51:05.036299099 +0000 UTC m=+76.540776697"
Sep 17 20:51:05.101940 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.101842    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rgxgj\" (UID: \"5696194e-5b60-4427-9147-1c42eb17a7ed\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:05.102079 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:05.102016    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:05.102129 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:05.102105    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls podName:5696194e-5b60-4427-9147-1c42eb17a7ed nodeName:}" failed. No retries permitted until 2026-09-17 20:51:09.10208358 +0000 UTC m=+80.606561159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-rgxgj" (UID: "5696194e-5b60-4427-9147-1c42eb17a7ed") : secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:05.140306 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.140272    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68747b9cb5-fnnrl"]
Sep 17 20:51:05.142178 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.142161    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.145183 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.145159    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vfk5t\""
Sep 17 20:51:05.145293 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.145193    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 20:51:05.145460 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.145441    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 20:51:05.145596 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.145583    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 20:51:05.152047 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.152025    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 20:51:05.156317 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.156296    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68747b9cb5-fnnrl"]
Sep 17 20:51:05.202510 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.202466    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acd16910-0f29-4e02-8cb9-d65894ee8d1a-installation-pull-secrets\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.202510 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.202512    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xwkw\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-kube-api-access-6xwkw\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.202786 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.202652    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acd16910-0f29-4e02-8cb9-d65894ee8d1a-trusted-ca\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.202786 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.202705    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-certificates\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.202786 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.202755    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acd16910-0f29-4e02-8cb9-d65894ee8d1a-ca-trust-extracted\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.202786 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.202773    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-bound-sa-token\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.202977 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.202792    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acd16910-0f29-4e02-8cb9-d65894ee8d1a-image-registry-private-configuration\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.202977 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.202868    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.304039 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.304002    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.304235 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.304056    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acd16910-0f29-4e02-8cb9-d65894ee8d1a-installation-pull-secrets\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.304235 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.304114    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xwkw\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-kube-api-access-6xwkw\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.304235 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:05.304182    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:51:05.304235 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.304188    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acd16910-0f29-4e02-8cb9-d65894ee8d1a-trusted-ca\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.304235 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:05.304198    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68747b9cb5-fnnrl: secret "image-registry-tls" not found
Sep 17 20:51:05.304470 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:05.304259    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls podName:acd16910-0f29-4e02-8cb9-d65894ee8d1a nodeName:}" failed. No retries permitted until 2026-09-17 20:51:05.804238776 +0000 UTC m=+77.308716372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls") pod "image-registry-68747b9cb5-fnnrl" (UID: "acd16910-0f29-4e02-8cb9-d65894ee8d1a") : secret "image-registry-tls" not found
Sep 17 20:51:05.304470 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.304391    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-certificates\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.304470 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.304458    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acd16910-0f29-4e02-8cb9-d65894ee8d1a-ca-trust-extracted\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.304652 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.304495    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-bound-sa-token\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.304652 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.304531    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acd16910-0f29-4e02-8cb9-d65894ee8d1a-image-registry-private-configuration\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.305255 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.305221    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acd16910-0f29-4e02-8cb9-d65894ee8d1a-ca-trust-extracted\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.305355 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.305221    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-certificates\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.306068 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.306035    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acd16910-0f29-4e02-8cb9-d65894ee8d1a-trusted-ca\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.307456 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.307436    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acd16910-0f29-4e02-8cb9-d65894ee8d1a-installation-pull-secrets\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.307751 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.307731    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acd16910-0f29-4e02-8cb9-d65894ee8d1a-image-registry-private-configuration\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.320134 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.317461    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-bound-sa-token\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.320134 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.318822    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xwkw\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-kube-api-access-6xwkw\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.808336 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:05.808291    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:05.808585 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:05.808437    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:51:05.808585 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:05.808453    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68747b9cb5-fnnrl: secret "image-registry-tls" not found
Sep 17 20:51:05.808585 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:05.808522    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls podName:acd16910-0f29-4e02-8cb9-d65894ee8d1a nodeName:}" failed. No retries permitted until 2026-09-17 20:51:06.808502677 +0000 UTC m=+78.312980268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls") pod "image-registry-68747b9cb5-fnnrl" (UID: "acd16910-0f29-4e02-8cb9-d65894ee8d1a") : secret "image-registry-tls" not found
Sep 17 20:51:06.058016 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.057975    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5hc26"]
Sep 17 20:51:06.061658 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.061637    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:06.063949 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.063928    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 20:51:06.064051 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.063999    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qm4s6\""
Sep 17 20:51:06.064051 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.064022    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 20:51:06.076601 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.076579    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5hc26"]
Sep 17 20:51:06.111993 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.111967    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:06.112163 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.112002    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:06.112163 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.112081    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-crio-socket\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:06.112163 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.112152    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-264mk\" (UniqueName: \"kubernetes.io/projected/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-api-access-264mk\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:06.112323 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.112183    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-data-volume\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:06.212581 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.212532    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:06.212581 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.212583    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:06.212808 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.212653    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-crio-socket\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:06.212808 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.212678    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-264mk\" (UniqueName: \"kubernetes.io/projected/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-api-access-264mk\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:06.212808 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:06.212686    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:06.212808 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.212761    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-crio-socket\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:06.212808 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:06.212772    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm podName:92cc791b-7e51-4256-9e52-0dbf7d51ee0f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:06.712752073 +0000 UTC m=+78.217229651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5hc26" (UID: "92cc791b-7e51-4256-9e52-0dbf7d51ee0f") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:06.212808 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.212696    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-data-volume\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:06.213072 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:06.212799    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:06.213072 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:06.212955    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls podName:92cc791b-7e51-4256-9e52-0dbf7d51ee0f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:06.71293279 +0000 UTC m=+78.217410364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5hc26" (UID: "92cc791b-7e51-4256-9e52-0dbf7d51ee0f") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:06.213072 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.213033    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-data-volume\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:06.221304 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.221276    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-264mk\" (UniqueName: \"kubernetes.io/projected/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-api-access-264mk\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:06.716736 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.716690    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:06.716918 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.716745    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:06.716918 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:06.716819    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:06.716918 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:06.716894    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm podName:92cc791b-7e51-4256-9e52-0dbf7d51ee0f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:07.716879073 +0000 UTC m=+79.221356661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5hc26" (UID: "92cc791b-7e51-4256-9e52-0dbf7d51ee0f") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:06.717025 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:06.716922    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:06.717025 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:06.716981    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls podName:92cc791b-7e51-4256-9e52-0dbf7d51ee0f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:07.716967691 +0000 UTC m=+79.221445265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5hc26" (UID: "92cc791b-7e51-4256-9e52-0dbf7d51ee0f") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:06.817915 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:06.817881    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:06.818075 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:06.818000    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:51:06.818075 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:06.818013    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68747b9cb5-fnnrl: secret "image-registry-tls" not found
Sep 17 20:51:06.818075 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:06.818067    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls podName:acd16910-0f29-4e02-8cb9-d65894ee8d1a nodeName:}" failed. No retries permitted until 2026-09-17 20:51:08.818053074 +0000 UTC m=+80.322530671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls") pod "image-registry-68747b9cb5-fnnrl" (UID: "acd16910-0f29-4e02-8cb9-d65894ee8d1a") : secret "image-registry-tls" not found
Sep 17 20:51:07.725360 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:07.725319    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:07.725360 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:07.725365    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:07.725810 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:07.725462    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:07.725810 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:07.725476    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:07.725810 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:07.725570    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm podName:92cc791b-7e51-4256-9e52-0dbf7d51ee0f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:09.725527664 +0000 UTC m=+81.230005244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5hc26" (UID: "92cc791b-7e51-4256-9e52-0dbf7d51ee0f") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:07.725810 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:07.725591    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls podName:92cc791b-7e51-4256-9e52-0dbf7d51ee0f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:09.725580901 +0000 UTC m=+81.230058479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5hc26" (UID: "92cc791b-7e51-4256-9e52-0dbf7d51ee0f") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:08.006194 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:08.006096    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n779j_0cc2bef5-b9e2-4a70-9905-f425604d5f3c/dns-node-resolver/0.log"
Sep 17 20:51:08.834360 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:08.834324    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:08.834781 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:08.834439    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:51:08.834781 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:08.834450    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68747b9cb5-fnnrl: secret "image-registry-tls" not found
Sep 17 20:51:08.834781 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:08.834502    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls podName:acd16910-0f29-4e02-8cb9-d65894ee8d1a nodeName:}" failed. No retries permitted until 2026-09-17 20:51:12.834488899 +0000 UTC m=+84.338966472 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls") pod "image-registry-68747b9cb5-fnnrl" (UID: "acd16910-0f29-4e02-8cb9-d65894ee8d1a") : secret "image-registry-tls" not found
Sep 17 20:51:09.035169 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:09.035125    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-kcmss\" (UID: \"964e7579-62e6-4dd0-92a1-05569d75d125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss"
Sep 17 20:51:09.035342 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:09.035280    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 20:51:09.035380 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:09.035343    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls podName:964e7579-62e6-4dd0-92a1-05569d75d125 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:17.035327693 +0000 UTC m=+88.539805267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-kcmss" (UID: "964e7579-62e6-4dd0-92a1-05569d75d125") : secret "samples-operator-tls" not found
Sep 17 20:51:09.136539 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:09.136456    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rgxgj\" (UID: \"5696194e-5b60-4427-9147-1c42eb17a7ed\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:09.136715 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:09.136594    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:09.136715 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:09.136651    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls podName:5696194e-5b60-4427-9147-1c42eb17a7ed nodeName:}" failed. No retries permitted until 2026-09-17 20:51:17.136636256 +0000 UTC m=+88.641113830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-rgxgj" (UID: "5696194e-5b60-4427-9147-1c42eb17a7ed") : secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:09.206467 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:09.206439    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qmv8f_d934ceba-229e-4206-bb5f-31b8e8af019f/node-ca/0.log"
Sep 17 20:51:09.742233 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:09.742184    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:09.742233 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:09.742237    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:09.742494 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:09.742330    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:09.742494 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:09.742350    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:09.742494 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:09.742402    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls podName:92cc791b-7e51-4256-9e52-0dbf7d51ee0f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:13.74238918 +0000 UTC m=+85.246866755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5hc26" (UID: "92cc791b-7e51-4256-9e52-0dbf7d51ee0f") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:09.742494 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:09.742415    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm podName:92cc791b-7e51-4256-9e52-0dbf7d51ee0f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:13.742409138 +0000 UTC m=+85.246886712 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5hc26" (UID: "92cc791b-7e51-4256-9e52-0dbf7d51ee0f") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:10.006915 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:10.006829    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f2bdn_07aff763-5b60-4dda-89de-f72e2896a715/serve-healthcheck-canary/0.log"
Sep 17 20:51:12.870281 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:12.870242    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:12.870695 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:12.870411    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:51:12.870695 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:12.870431    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68747b9cb5-fnnrl: secret "image-registry-tls" not found
Sep 17 20:51:12.870695 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:12.870488    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls podName:acd16910-0f29-4e02-8cb9-d65894ee8d1a nodeName:}" failed. No retries permitted until 2026-09-17 20:51:20.87047366 +0000 UTC m=+92.374951254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls") pod "image-registry-68747b9cb5-fnnrl" (UID: "acd16910-0f29-4e02-8cb9-d65894ee8d1a") : secret "image-registry-tls" not found
Sep 17 20:51:13.777009 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:13.776968    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:13.777009 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:13.777006    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:13.777208 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:13.777093    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:13.777208 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:13.777106    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:13.777208 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:13.777158    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls podName:92cc791b-7e51-4256-9e52-0dbf7d51ee0f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:21.77714441 +0000 UTC m=+93.281621998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5hc26" (UID: "92cc791b-7e51-4256-9e52-0dbf7d51ee0f") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:13.777208 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:13.777170    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm podName:92cc791b-7e51-4256-9e52-0dbf7d51ee0f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:21.777163771 +0000 UTC m=+93.281641345 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5hc26" (UID: "92cc791b-7e51-4256-9e52-0dbf7d51ee0f") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:17.105001 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:17.104960    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-kcmss\" (UID: \"964e7579-62e6-4dd0-92a1-05569d75d125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss"
Sep 17 20:51:17.107517 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:17.107497    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/964e7579-62e6-4dd0-92a1-05569d75d125-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-kcmss\" (UID: \"964e7579-62e6-4dd0-92a1-05569d75d125\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss"
Sep 17 20:51:17.205708 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:17.205668    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rgxgj\" (UID: \"5696194e-5b60-4427-9147-1c42eb17a7ed\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:17.205889 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:17.205825    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:17.205930 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:17.205894    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls podName:5696194e-5b60-4427-9147-1c42eb17a7ed nodeName:}" failed. No retries permitted until 2026-09-17 20:51:33.205877436 +0000 UTC m=+104.710355009 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-rgxgj" (UID: "5696194e-5b60-4427-9147-1c42eb17a7ed") : secret "cluster-monitoring-operator-tls" not found
Sep 17 20:51:17.223368 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:17.223339    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss"
Sep 17 20:51:17.491917 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:17.491845    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss"]
Sep 17 20:51:18.021382 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:18.021339    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss" event={"ID":"964e7579-62e6-4dd0-92a1-05569d75d125","Type":"ContainerStarted","Data":"8553c3cd016958659c9713d9948f22a71d0c794668dcf3b085a66ba2f93e3c5c"}
Sep 17 20:51:19.026252 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:19.026219    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss" event={"ID":"964e7579-62e6-4dd0-92a1-05569d75d125","Type":"ContainerStarted","Data":"edf7211281c002c2b7dd3b3b7ca4aec16e81d2b20c018728c95a13fb2a5380c5"}
Sep 17 20:51:20.030812 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:20.030780    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss" event={"ID":"964e7579-62e6-4dd0-92a1-05569d75d125","Type":"ContainerStarted","Data":"4095af864fcab62ac8bb056f176e35e3e9d6bde932ff66cab909c39a26b2102a"}
Sep 17 20:51:20.052664 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:20.052616    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-kcmss" podStartSLOduration=17.634755699 podStartE2EDuration="19.05259863s" totalImagesPullingTime="1.417842931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:20.051168337 +0000 UTC m=+91.555645933" watchObservedRunningTime="2026-09-17 20:51:20.05259863 +0000 UTC m=+91.557076226"
Sep 17 20:51:20.933161 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:20.933107    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:20.935608 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:20.935588    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls\") pod \"image-registry-68747b9cb5-fnnrl\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") " pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:21.053541 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:21.053513    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:21.328584 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:21.328533    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68747b9cb5-fnnrl"]
Sep 17 20:51:21.331418 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:21.331389    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd16910_0f29_4e02_8cb9_d65894ee8d1a.slice/crio-23349403e766554843f978917c6fc67d14a968ee8c0d6a7411ceef1575e4d381 WatchSource:0}: Error finding container 23349403e766554843f978917c6fc67d14a968ee8c0d6a7411ceef1575e4d381: Status 404 returned error can't find the container with id 23349403e766554843f978917c6fc67d14a968ee8c0d6a7411ceef1575e4d381
Sep 17 20:51:21.840981 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:21.840943    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:21.841248 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:21.840988    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:21.841248 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:21.841101    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:21.841248 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:21.841168    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls podName:92cc791b-7e51-4256-9e52-0dbf7d51ee0f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:37.841153972 +0000 UTC m=+109.345631559 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5hc26" (UID: "92cc791b-7e51-4256-9e52-0dbf7d51ee0f") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:21.841619 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:21.841595    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:22.038860 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:22.038824    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl" event={"ID":"acd16910-0f29-4e02-8cb9-d65894ee8d1a","Type":"ContainerStarted","Data":"dbdac0970f7056d4356606158303afdac804cfe2e669e46a1b4069847ecf3551"}
Sep 17 20:51:22.039027 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:22.038873    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl" event={"ID":"acd16910-0f29-4e02-8cb9-d65894ee8d1a","Type":"ContainerStarted","Data":"23349403e766554843f978917c6fc67d14a968ee8c0d6a7411ceef1575e4d381"}
Sep 17 20:51:22.039027 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:22.038954    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:22.064086 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:22.064038    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl" podStartSLOduration=17.064021536 podStartE2EDuration="17.064021536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:22.062833209 +0000 UTC m=+93.567310818" watchObservedRunningTime="2026-09-17 20:51:22.064021536 +0000 UTC m=+93.568499134"
Sep 17 20:51:28.393571 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:28.393510    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:51:28.395962 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:28.395943    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/325ad1ea-7622-490d-82c3-ff73e8ff5dda-metrics-tls\") pod \"dns-default-mp5gn\" (UID: \"325ad1ea-7622-490d-82c3-ff73e8ff5dda\") " pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:51:28.425809 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:28.425781    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k4498\""
Sep 17 20:51:28.431459 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:28.431445    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:51:28.713044 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:28.712962    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mp5gn"]
Sep 17 20:51:28.715771 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:28.715734    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325ad1ea_7622_490d_82c3_ff73e8ff5dda.slice/crio-128f1a4ff2638db0f59bb0dd491eea7be8b80fa479fdfc4db46f0de487fc3a63 WatchSource:0}: Error finding container 128f1a4ff2638db0f59bb0dd491eea7be8b80fa479fdfc4db46f0de487fc3a63: Status 404 returned error can't find the container with id 128f1a4ff2638db0f59bb0dd491eea7be8b80fa479fdfc4db46f0de487fc3a63
Sep 17 20:51:29.061031 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.060985    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mp5gn" event={"ID":"325ad1ea-7622-490d-82c3-ff73e8ff5dda","Type":"ContainerStarted","Data":"128f1a4ff2638db0f59bb0dd491eea7be8b80fa479fdfc4db46f0de487fc3a63"}
Sep 17 20:51:29.089810 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.089781    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-xlfkt"]
Sep 17 20:51:29.093760 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.093743    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-8g7bq"]
Sep 17 20:51:29.093879 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.093860    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-xlfkt"
Sep 17 20:51:29.096485 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.096458    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8g7bq"
Sep 17 20:51:29.096629 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.096572    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-x25sz\""
Sep 17 20:51:29.096629 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.096584    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 20:51:29.097661 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.097649    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 20:51:29.099064 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.099046    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xblnq\""
Sep 17 20:51:29.111765 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.111744    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-xlfkt"]
Sep 17 20:51:29.112654 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.112633    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-8g7bq"]
Sep 17 20:51:29.182740 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.182706    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68747b9cb5-fnnrl"]
Sep 17 20:51:29.199818 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.199778    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bdc1fd31-eb35-4d76-9364-c26bacdcf59f-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-xlfkt\" (UID: \"bdc1fd31-eb35-4d76-9364-c26bacdcf59f\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-xlfkt"
Sep 17 20:51:29.199973 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.199931    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bdc1fd31-eb35-4d76-9364-c26bacdcf59f-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-xlfkt\" (UID: \"bdc1fd31-eb35-4d76-9364-c26bacdcf59f\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-xlfkt"
Sep 17 20:51:29.200035 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.199990    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdl7z\" (UniqueName: \"kubernetes.io/projected/fb1ed212-5c4f-4b71-b7f7-6057ba0eb12b-kube-api-access-kdl7z\") pod \"network-check-source-7cbc45fddc-8g7bq\" (UID: \"fb1ed212-5c4f-4b71-b7f7-6057ba0eb12b\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8g7bq"
Sep 17 20:51:29.300689 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.300654    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdl7z\" (UniqueName: \"kubernetes.io/projected/fb1ed212-5c4f-4b71-b7f7-6057ba0eb12b-kube-api-access-kdl7z\") pod \"network-check-source-7cbc45fddc-8g7bq\" (UID: \"fb1ed212-5c4f-4b71-b7f7-6057ba0eb12b\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8g7bq"
Sep 17 20:51:29.300877 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.300713    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bdc1fd31-eb35-4d76-9364-c26bacdcf59f-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-xlfkt\" (UID: \"bdc1fd31-eb35-4d76-9364-c26bacdcf59f\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-xlfkt"
Sep 17 20:51:29.300877 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.300793    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bdc1fd31-eb35-4d76-9364-c26bacdcf59f-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-xlfkt\" (UID: \"bdc1fd31-eb35-4d76-9364-c26bacdcf59f\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-xlfkt"
Sep 17 20:51:29.301572 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.301521    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bdc1fd31-eb35-4d76-9364-c26bacdcf59f-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-xlfkt\" (UID: \"bdc1fd31-eb35-4d76-9364-c26bacdcf59f\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-xlfkt"
Sep 17 20:51:29.303768 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.303744    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bdc1fd31-eb35-4d76-9364-c26bacdcf59f-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-xlfkt\" (UID: \"bdc1fd31-eb35-4d76-9364-c26bacdcf59f\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-xlfkt"
Sep 17 20:51:29.315084 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.315026    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdl7z\" (UniqueName: \"kubernetes.io/projected/fb1ed212-5c4f-4b71-b7f7-6057ba0eb12b-kube-api-access-kdl7z\") pod \"network-check-source-7cbc45fddc-8g7bq\" (UID: \"fb1ed212-5c4f-4b71-b7f7-6057ba0eb12b\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8g7bq"
Sep 17 20:51:29.403482 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.403457    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-xlfkt"
Sep 17 20:51:29.408643 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.408621    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8g7bq"
Sep 17 20:51:29.681286 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.680857    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-xlfkt"]
Sep 17 20:51:29.684240 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:29.684209    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc1fd31_eb35_4d76_9364_c26bacdcf59f.slice/crio-3c7abe0cde79478142dfaba57f068feea4a53cf894463c03565e8ef687425b71 WatchSource:0}: Error finding container 3c7abe0cde79478142dfaba57f068feea4a53cf894463c03565e8ef687425b71: Status 404 returned error can't find the container with id 3c7abe0cde79478142dfaba57f068feea4a53cf894463c03565e8ef687425b71
Sep 17 20:51:29.700008 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:29.699982    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-8g7bq"]
Sep 17 20:51:29.703317 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:29.703293    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1ed212_5c4f_4b71_b7f7_6057ba0eb12b.slice/crio-0baa67c6c30dafa2064e9ab06a825cb8c98c09ade011bedf72506c8268ae35da WatchSource:0}: Error finding container 0baa67c6c30dafa2064e9ab06a825cb8c98c09ade011bedf72506c8268ae35da: Status 404 returned error can't find the container with id 0baa67c6c30dafa2064e9ab06a825cb8c98c09ade011bedf72506c8268ae35da
Sep 17 20:51:30.067030 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:30.066975    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8g7bq" event={"ID":"fb1ed212-5c4f-4b71-b7f7-6057ba0eb12b","Type":"ContainerStarted","Data":"a6232ec8ca5789b9351fb90fd5ad6a4bce1d2117c6e3c8a5e30bf7317aa14e54"}
Sep 17 20:51:30.067213 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:30.067098    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8g7bq" event={"ID":"fb1ed212-5c4f-4b71-b7f7-6057ba0eb12b","Type":"ContainerStarted","Data":"0baa67c6c30dafa2064e9ab06a825cb8c98c09ade011bedf72506c8268ae35da"}
Sep 17 20:51:30.068326 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:30.068298    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-xlfkt" event={"ID":"bdc1fd31-eb35-4d76-9364-c26bacdcf59f","Type":"ContainerStarted","Data":"3c7abe0cde79478142dfaba57f068feea4a53cf894463c03565e8ef687425b71"}
Sep 17 20:51:30.088128 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:30.088079    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8g7bq" podStartSLOduration=1.088062141 podStartE2EDuration="1.088062141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:30.085743831 +0000 UTC m=+101.590221427" watchObservedRunningTime="2026-09-17 20:51:30.088062141 +0000 UTC m=+101.592539736"
Sep 17 20:51:31.073271 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:31.073236    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mp5gn" event={"ID":"325ad1ea-7622-490d-82c3-ff73e8ff5dda","Type":"ContainerStarted","Data":"76603d3ff7c5e704950601d4299ffd0807d667864504685ca73190a4238e5bcb"}
Sep 17 20:51:31.073742 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:31.073289    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mp5gn" event={"ID":"325ad1ea-7622-490d-82c3-ff73e8ff5dda","Type":"ContainerStarted","Data":"bbc61d261f3b1255887ee75a2474cdecab34f6a8fd8eb0e16fa541c74e30c9e9"}
Sep 17 20:51:31.094163 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:31.094120    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mp5gn" podStartSLOduration=65.459528885 podStartE2EDuration="1m7.094102379s" totalImagesPullingTime="1.634573494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:31.092457261 +0000 UTC m=+102.596934868" watchObservedRunningTime="2026-09-17 20:51:31.094102379 +0000 UTC m=+102.598579974"
Sep 17 20:51:32.078421 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:32.078377    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-xlfkt" event={"ID":"bdc1fd31-eb35-4d76-9364-c26bacdcf59f","Type":"ContainerStarted","Data":"d073ab11e17803b72c9db4c37bc6304221d24f33f02525d3e55af7660c23d25b"}
Sep 17 20:51:32.078841 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:32.078611    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:51:32.108310 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:32.108252    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-xlfkt" podStartSLOduration=1.715482868 podStartE2EDuration="3.108231443s" totalImagesPullingTime="1.392748575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:32.10677761 +0000 UTC m=+103.611255206" watchObservedRunningTime="2026-09-17 20:51:32.108231443 +0000 UTC m=+103.612709039"
Sep 17 20:51:32.973680 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:32.973648    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-csv4n"
Sep 17 20:51:33.233975 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:33.233886    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rgxgj\" (UID: \"5696194e-5b60-4427-9147-1c42eb17a7ed\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:33.236404 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:33.236376    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5696194e-5b60-4427-9147-1c42eb17a7ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rgxgj\" (UID: \"5696194e-5b60-4427-9147-1c42eb17a7ed\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:33.534141 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:33.534057    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"
Sep 17 20:51:33.837325 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:33.837295    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj"]
Sep 17 20:51:33.840137 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:33.840109    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5696194e_5b60_4427_9147_1c42eb17a7ed.slice/crio-e23e2729d9c3ef87cf273bbf55f2002e2417faa9e1555421740c196e450b0647 WatchSource:0}: Error finding container e23e2729d9c3ef87cf273bbf55f2002e2417faa9e1555421740c196e450b0647: Status 404 returned error can't find the container with id e23e2729d9c3ef87cf273bbf55f2002e2417faa9e1555421740c196e450b0647
Sep 17 20:51:34.085571 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:34.085515    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj" event={"ID":"5696194e-5b60-4427-9147-1c42eb17a7ed","Type":"ContainerStarted","Data":"e23e2729d9c3ef87cf273bbf55f2002e2417faa9e1555421740c196e450b0647"}
Sep 17 20:51:35.972992 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:35.972956    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bb8bfdffd-24zth"]
Sep 17 20:51:35.982726 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:35.982700    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:35.989699 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:35.989160    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 20:51:35.989699 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:35.989442    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 20:51:35.990134 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:35.990114    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 20:51:35.990253 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:35.990117    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 20:51:35.990253 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:35.990192    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 20:51:35.990368 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:35.990148    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 20:51:35.990428 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:35.990395    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xqlsk\""
Sep 17 20:51:35.990488 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:35.990465    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 20:51:35.990770 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:35.990745    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bb8bfdffd-24zth"]
Sep 17 20:51:36.057358 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.057323    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfspv\" (UniqueName: \"kubernetes.io/projected/9e7c0e30-282e-41d9-90bc-39f26a48444a-kube-api-access-qfspv\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.057358 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.057360    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-service-ca\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.057597 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.057384    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-oauth-serving-cert\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.057597 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.057442    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-oauth-config\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.057597 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.057475    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-config\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.057597 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.057498    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-serving-cert\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.094415 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.094331    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj" event={"ID":"5696194e-5b60-4427-9147-1c42eb17a7ed","Type":"ContainerStarted","Data":"ecce9a99c3805b95c8a60eaf175ac84da46ef07809dc2615c1929d79bdb5cfa6"}
Sep 17 20:51:36.117591 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.115118    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rgxgj" podStartSLOduration=33.118745108 podStartE2EDuration="35.115095983s" totalImagesPullingTime="1.996350875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:36.112790075 +0000 UTC m=+107.617267671" watchObservedRunningTime="2026-09-17 20:51:36.115095983 +0000 UTC m=+107.619573582"
Sep 17 20:51:36.157918 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.157881    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-config\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.158119 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.157930    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-serving-cert\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.158119 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.158091    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfspv\" (UniqueName: \"kubernetes.io/projected/9e7c0e30-282e-41d9-90bc-39f26a48444a-kube-api-access-qfspv\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.158236 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.158150    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-service-ca\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.158236 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.158209    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-oauth-serving-cert\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.158346 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.158244    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-oauth-config\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.158768 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.158714    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-config\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.158996 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.158943    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-service-ca\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.159108 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.159046    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-oauth-serving-cert\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.160951 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.160906    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-serving-cert\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.161224 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.161207    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-oauth-config\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.173750 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.173729    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfspv\" (UniqueName: \"kubernetes.io/projected/9e7c0e30-282e-41d9-90bc-39f26a48444a-kube-api-access-qfspv\") pod \"console-5bb8bfdffd-24zth\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") " pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.292784 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.292740    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:36.402643 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.402612    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2chxs"]
Sep 17 20:51:36.434120 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.434080    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2chxs"]
Sep 17 20:51:36.434315 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.434221    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2chxs"
Sep 17 20:51:36.437668 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.437613    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-qt2bk\""
Sep 17 20:51:36.437668 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.437621    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 20:51:36.561415 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.561366    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/51c1c187-47e0-48a8-aeb8-1f46821144ac-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-2chxs\" (UID: \"51c1c187-47e0-48a8-aeb8-1f46821144ac\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2chxs"
Sep 17 20:51:36.603000 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.602977    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bb8bfdffd-24zth"]
Sep 17 20:51:36.605697 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:36.605673    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7c0e30_282e_41d9_90bc_39f26a48444a.slice/crio-928922813ee2fab85316ed59cf79323e8be8a7f30c508922d6958bfc82aca673 WatchSource:0}: Error finding container 928922813ee2fab85316ed59cf79323e8be8a7f30c508922d6958bfc82aca673: Status 404 returned error can't find the container with id 928922813ee2fab85316ed59cf79323e8be8a7f30c508922d6958bfc82aca673
Sep 17 20:51:36.662903 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.662816    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/51c1c187-47e0-48a8-aeb8-1f46821144ac-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-2chxs\" (UID: \"51c1c187-47e0-48a8-aeb8-1f46821144ac\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2chxs"
Sep 17 20:51:36.665601 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.665576    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/51c1c187-47e0-48a8-aeb8-1f46821144ac-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-2chxs\" (UID: \"51c1c187-47e0-48a8-aeb8-1f46821144ac\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2chxs"
Sep 17 20:51:36.744125 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:36.744092    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2chxs"
Sep 17 20:51:37.028820 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:37.028785    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2chxs"]
Sep 17 20:51:37.030530 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:37.030501    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c1c187_47e0_48a8_aeb8_1f46821144ac.slice/crio-87220cbf841c53e3198a14a73b5196e8b691572d45190a68a92e1acab68328cf WatchSource:0}: Error finding container 87220cbf841c53e3198a14a73b5196e8b691572d45190a68a92e1acab68328cf: Status 404 returned error can't find the container with id 87220cbf841c53e3198a14a73b5196e8b691572d45190a68a92e1acab68328cf
Sep 17 20:51:37.099997 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:37.099946    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2chxs" event={"ID":"51c1c187-47e0-48a8-aeb8-1f46821144ac","Type":"ContainerStarted","Data":"87220cbf841c53e3198a14a73b5196e8b691572d45190a68a92e1acab68328cf"}
Sep 17 20:51:37.101381 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:37.101351    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bb8bfdffd-24zth" event={"ID":"9e7c0e30-282e-41d9-90bc-39f26a48444a","Type":"ContainerStarted","Data":"928922813ee2fab85316ed59cf79323e8be8a7f30c508922d6958bfc82aca673"}
Sep 17 20:51:37.873309 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:37.873256    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:37.876418 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:37.876387    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92cc791b-7e51-4256-9e52-0dbf7d51ee0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5hc26\" (UID: \"92cc791b-7e51-4256-9e52-0dbf7d51ee0f\") " pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:38.171173 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.171088    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5hc26"
Sep 17 20:51:38.355630 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.355589    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c5cdb566b-p7xlh"]
Sep 17 20:51:38.384918 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.384883    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5cdb566b-p7xlh"]
Sep 17 20:51:38.385105 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.384993    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.394830 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.394616    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 20:51:38.478790 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.478701    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-oauth-config\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.478790 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.478746    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-service-ca\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.479024 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.478774    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-config\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.479024 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.478843    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-trusted-ca-bundle\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.479024 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.478872    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-oauth-serving-cert\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.479024 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.478965    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-serving-cert\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.479263 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.479022    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp76z\" (UniqueName: \"kubernetes.io/projected/bcab22c0-5a7b-45db-b464-657a40d83e0e-kube-api-access-tp76z\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.579951 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.579899    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-oauth-config\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.580143 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.579961    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-service-ca\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.580143 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.579992    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-config\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.580143 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.580020    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-trusted-ca-bundle\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.580143 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.580043    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-oauth-serving-cert\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.580143 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.580124    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-serving-cert\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.580390 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.580168    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp76z\" (UniqueName: \"kubernetes.io/projected/bcab22c0-5a7b-45db-b464-657a40d83e0e-kube-api-access-tp76z\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.581069 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.581044    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-config\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.581200 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.581071    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-oauth-serving-cert\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.581268 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.581198    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-service-ca\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.581715 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.581688    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-trusted-ca-bundle\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.583440 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.583417    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-oauth-config\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.583567 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.583417    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-serving-cert\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.589702 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.589657    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp76z\" (UniqueName: \"kubernetes.io/projected/bcab22c0-5a7b-45db-b464-657a40d83e0e-kube-api-access-tp76z\") pod \"console-5c5cdb566b-p7xlh\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") " pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:38.697524 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:38.697490    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:39.188443 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:39.188407    2571 patch_prober.go:28] interesting pod/image-registry-68747b9cb5-fnnrl container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 20:51:39.188918 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:39.188470    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl" podUID="acd16910-0f29-4e02-8cb9-d65894ee8d1a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 20:51:41.428258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:41.428226    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5cdb566b-p7xlh"]
Sep 17 20:51:41.430056 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:41.430025    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcab22c0_5a7b_45db_b464_657a40d83e0e.slice/crio-995aeae46fe15a90fa4e587f64e1efb0da973519cee33653bddb3778ca041902 WatchSource:0}: Error finding container 995aeae46fe15a90fa4e587f64e1efb0da973519cee33653bddb3778ca041902: Status 404 returned error can't find the container with id 995aeae46fe15a90fa4e587f64e1efb0da973519cee33653bddb3778ca041902
Sep 17 20:51:41.430324 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:41.430305    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5hc26"]
Sep 17 20:51:41.432675 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:41.432654    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cc791b_7e51_4256_9e52_0dbf7d51ee0f.slice/crio-6ab02bc48d3996f8df4891be20b6303742ec5ce51d5fb531cb19da4930b3a147 WatchSource:0}: Error finding container 6ab02bc48d3996f8df4891be20b6303742ec5ce51d5fb531cb19da4930b3a147: Status 404 returned error can't find the container with id 6ab02bc48d3996f8df4891be20b6303742ec5ce51d5fb531cb19da4930b3a147
Sep 17 20:51:42.083681 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:42.083658    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mp5gn"
Sep 17 20:51:42.126614 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:42.126583    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5cdb566b-p7xlh" event={"ID":"bcab22c0-5a7b-45db-b464-657a40d83e0e","Type":"ContainerStarted","Data":"df89be0a78643f640b542c84cc418cf3960ed1d0c701f722a8e249b199c447db"}
Sep 17 20:51:42.126707 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:42.126640    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5cdb566b-p7xlh" event={"ID":"bcab22c0-5a7b-45db-b464-657a40d83e0e","Type":"ContainerStarted","Data":"995aeae46fe15a90fa4e587f64e1efb0da973519cee33653bddb3778ca041902"}
Sep 17 20:51:42.128803 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:42.128772    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bb8bfdffd-24zth" event={"ID":"9e7c0e30-282e-41d9-90bc-39f26a48444a","Type":"ContainerStarted","Data":"9dcf1151a4358f0c53b9c18702953cbe1f779446f75f6ca2e81e6f2c57d3b418"}
Sep 17 20:51:42.132319 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:42.132294    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2chxs" event={"ID":"51c1c187-47e0-48a8-aeb8-1f46821144ac","Type":"ContainerStarted","Data":"5b29b0ab1be0fc2d1da9cc9c6afaf5499633bc0bb7d9741203a247b931f33032"}
Sep 17 20:51:42.132488 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:42.132457    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2chxs"
Sep 17 20:51:42.135229 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:42.135200    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5hc26" event={"ID":"92cc791b-7e51-4256-9e52-0dbf7d51ee0f","Type":"ContainerStarted","Data":"45936204aeceab730dc7f020a22014579c8cf6d4262d0d74df635c176caeb258"}
Sep 17 20:51:42.135344 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:42.135252    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5hc26" event={"ID":"92cc791b-7e51-4256-9e52-0dbf7d51ee0f","Type":"ContainerStarted","Data":"6ab02bc48d3996f8df4891be20b6303742ec5ce51d5fb531cb19da4930b3a147"}
Sep 17 20:51:42.138271 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:42.138169    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2chxs"
Sep 17 20:51:42.155016 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:42.154969    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c5cdb566b-p7xlh" podStartSLOduration=4.154955286 podStartE2EDuration="4.154955286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:42.15464715 +0000 UTC m=+113.659124749" watchObservedRunningTime="2026-09-17 20:51:42.154955286 +0000 UTC m=+113.659432876"
Sep 17 20:51:42.178249 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:42.178196    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2chxs" podStartSLOduration=2.057169812 podStartE2EDuration="6.17817813s" totalImagesPullingTime="4.121008318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:42.176986943 +0000 UTC m=+113.681464541" watchObservedRunningTime="2026-09-17 20:51:42.17817813 +0000 UTC m=+113.682655716"
Sep 17 20:51:42.207530 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:42.207480    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bb8bfdffd-24zth" podStartSLOduration=2.626961658 podStartE2EDuration="7.207463848s" totalImagesPullingTime="4.58050219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:42.207010832 +0000 UTC m=+113.711488428" watchObservedRunningTime="2026-09-17 20:51:42.207463848 +0000 UTC m=+113.711941443"
Sep 17 20:51:43.141494 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:43.141450    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5hc26" event={"ID":"92cc791b-7e51-4256-9e52-0dbf7d51ee0f","Type":"ContainerStarted","Data":"1593449dfdd14766fa467498b8b002571d61e16185e8624698cd430ef2e58ef6"}
Sep 17 20:51:44.146523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:44.146486    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5hc26" event={"ID":"92cc791b-7e51-4256-9e52-0dbf7d51ee0f","Type":"ContainerStarted","Data":"2531a34b0b2aee41a8291517559c7c4e618f48a58e114b5a7b374740caec0cbf"}
Sep 17 20:51:44.168566 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:44.168506    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5hc26" podStartSLOduration=36.391132405 podStartE2EDuration="38.168490279s" totalImagesPullingTime="1.777357874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:51:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:44.167376427 +0000 UTC m=+115.671854023" watchObservedRunningTime="2026-09-17 20:51:44.168490279 +0000 UTC m=+115.672967879"
Sep 17 20:51:46.293421 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:46.293369    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:46.293830 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:46.293442    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:46.298359 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:46.298326    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:47.165348 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:47.165313    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:51:48.698395 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:48.698360    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:48.698815 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:48.698456    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:48.703901 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:48.703874    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:49.170316 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:49.170278    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:51:49.187594 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:49.187574    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:49.226490 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:49.226458    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bb8bfdffd-24zth"]
Sep 17 20:51:51.817363 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.817329    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"]
Sep 17 20:51:51.820258 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.820238    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"
Sep 17 20:51:51.823695 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.823674    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-767g6\""
Sep 17 20:51:51.823795 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.823676    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 20:51:51.823795 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.823678    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 20:51:51.823795 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.823712    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 20:51:51.828686 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.828668    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"]
Sep 17 20:51:51.830800 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.830784    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:51.833433 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.833414    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 20:51:51.833732 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.833719    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-m6wmc\""
Sep 17 20:51:51.834176 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.834161    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 20:51:51.836485 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.836464    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 20:51:51.837195 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.837176    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"]
Sep 17 20:51:51.848840 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.848814    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dbcl5"]
Sep 17 20:51:51.851159 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.851141    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:51.852254 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.852232    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"]
Sep 17 20:51:51.853822 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.853788    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pgb5d\""
Sep 17 20:51:51.853986 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.853958    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 20:51:51.854088 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.854076    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 20:51:51.854401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.854378    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 20:51:51.991419 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991385    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef08f252-a2f1-45ee-94a9-bb148e841f90-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:51.991419 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991417    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxqf4\" (UniqueName: \"kubernetes.io/projected/872e9212-3c43-4ef7-94aa-b27904af77c0-kube-api-access-nxqf4\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:51.991653 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991437    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ef08f252-a2f1-45ee-94a9-bb148e841f90-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:51.991653 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991457    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/872e9212-3c43-4ef7-94aa-b27904af77c0-node-exporter-tls\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:51.991653 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991541    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gknl\" (UniqueName: \"kubernetes.io/projected/ef08f252-a2f1-45ee-94a9-bb148e841f90-kube-api-access-4gknl\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:51.991653 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991601    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/872e9212-3c43-4ef7-94aa-b27904af77c0-root\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:51.991653 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991631    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef08f252-a2f1-45ee-94a9-bb148e841f90-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:51.991847 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991657    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41e43ece-8336-4b4c-9355-feaa49cd2d2c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-ksvcr\" (UID: \"41e43ece-8336-4b4c-9355-feaa49cd2d2c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"
Sep 17 20:51:51.991847 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991674    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/872e9212-3c43-4ef7-94aa-b27904af77c0-sys\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:51.991847 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991710    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/872e9212-3c43-4ef7-94aa-b27904af77c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:51.991847 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991772    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/872e9212-3c43-4ef7-94aa-b27904af77c0-node-exporter-textfile\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:51.991847 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991790    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcldx\" (UniqueName: \"kubernetes.io/projected/41e43ece-8336-4b4c-9355-feaa49cd2d2c-kube-api-access-hcldx\") pod \"openshift-state-metrics-86c5bc79d6-ksvcr\" (UID: \"41e43ece-8336-4b4c-9355-feaa49cd2d2c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"
Sep 17 20:51:51.991847 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991815    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/872e9212-3c43-4ef7-94aa-b27904af77c0-node-exporter-wtmp\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:51.991847 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991831    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef08f252-a2f1-45ee-94a9-bb148e841f90-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:51.991847 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991848    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/872e9212-3c43-4ef7-94aa-b27904af77c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:51.992110 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991872    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41e43ece-8336-4b4c-9355-feaa49cd2d2c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-ksvcr\" (UID: \"41e43ece-8336-4b4c-9355-feaa49cd2d2c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"
Sep 17 20:51:51.992110 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991937    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ef08f252-a2f1-45ee-94a9-bb148e841f90-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:51.992110 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991953    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/872e9212-3c43-4ef7-94aa-b27904af77c0-metrics-client-ca\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:51.992110 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:51.991989    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41e43ece-8336-4b4c-9355-feaa49cd2d2c-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-ksvcr\" (UID: \"41e43ece-8336-4b4c-9355-feaa49cd2d2c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"
Sep 17 20:51:52.092887 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.092808    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/872e9212-3c43-4ef7-94aa-b27904af77c0-node-exporter-textfile\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.092887 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.092850    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcldx\" (UniqueName: \"kubernetes.io/projected/41e43ece-8336-4b4c-9355-feaa49cd2d2c-kube-api-access-hcldx\") pod \"openshift-state-metrics-86c5bc79d6-ksvcr\" (UID: \"41e43ece-8336-4b4c-9355-feaa49cd2d2c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"
Sep 17 20:51:52.093098 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.092985    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/872e9212-3c43-4ef7-94aa-b27904af77c0-node-exporter-wtmp\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.093098 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093020    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef08f252-a2f1-45ee-94a9-bb148e841f90-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:52.093098 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093041    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/872e9212-3c43-4ef7-94aa-b27904af77c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.093098 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093069    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41e43ece-8336-4b4c-9355-feaa49cd2d2c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-ksvcr\" (UID: \"41e43ece-8336-4b4c-9355-feaa49cd2d2c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"
Sep 17 20:51:52.093297 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093116    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ef08f252-a2f1-45ee-94a9-bb148e841f90-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:52.093297 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093138    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/872e9212-3c43-4ef7-94aa-b27904af77c0-metrics-client-ca\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.093297 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093182    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/872e9212-3c43-4ef7-94aa-b27904af77c0-node-exporter-wtmp\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.093297 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093193    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41e43ece-8336-4b4c-9355-feaa49cd2d2c-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-ksvcr\" (UID: \"41e43ece-8336-4b4c-9355-feaa49cd2d2c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"
Sep 17 20:51:52.093297 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093264    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef08f252-a2f1-45ee-94a9-bb148e841f90-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:52.093297 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093290    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqf4\" (UniqueName: \"kubernetes.io/projected/872e9212-3c43-4ef7-94aa-b27904af77c0-kube-api-access-nxqf4\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.093643 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093318    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ef08f252-a2f1-45ee-94a9-bb148e841f90-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:52.093643 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093343    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/872e9212-3c43-4ef7-94aa-b27904af77c0-node-exporter-tls\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.093643 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093381    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gknl\" (UniqueName: \"kubernetes.io/projected/ef08f252-a2f1-45ee-94a9-bb148e841f90-kube-api-access-4gknl\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:52.093643 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:52.093405    2571 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 20:51:52.093643 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093420    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/872e9212-3c43-4ef7-94aa-b27904af77c0-root\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.093643 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093445    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef08f252-a2f1-45ee-94a9-bb148e841f90-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:52.093643 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:52.093470    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef08f252-a2f1-45ee-94a9-bb148e841f90-kube-state-metrics-tls podName:ef08f252-a2f1-45ee-94a9-bb148e841f90 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:52.593450391 +0000 UTC m=+124.097927992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ef08f252-a2f1-45ee-94a9-bb148e841f90-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-x4dbc" (UID: "ef08f252-a2f1-45ee-94a9-bb148e841f90") : secret "kube-state-metrics-tls" not found
Sep 17 20:51:52.093643 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093513    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41e43ece-8336-4b4c-9355-feaa49cd2d2c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-ksvcr\" (UID: \"41e43ece-8336-4b4c-9355-feaa49cd2d2c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"
Sep 17 20:51:52.093643 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093543    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/872e9212-3c43-4ef7-94aa-b27904af77c0-sys\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.093643 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093603    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/872e9212-3c43-4ef7-94aa-b27904af77c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.093643 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093608    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ef08f252-a2f1-45ee-94a9-bb148e841f90-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:52.094204 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093673    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/872e9212-3c43-4ef7-94aa-b27904af77c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.094204 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.093749    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/872e9212-3c43-4ef7-94aa-b27904af77c0-sys\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.094204 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.094126    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41e43ece-8336-4b4c-9355-feaa49cd2d2c-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-ksvcr\" (UID: \"41e43ece-8336-4b4c-9355-feaa49cd2d2c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"
Sep 17 20:51:52.094204 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.094187    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef08f252-a2f1-45ee-94a9-bb148e841f90-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:52.094471 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.094278    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/872e9212-3c43-4ef7-94aa-b27904af77c0-node-exporter-textfile\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.094471 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.094303    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/872e9212-3c43-4ef7-94aa-b27904af77c0-root\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.094716 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.094695    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/872e9212-3c43-4ef7-94aa-b27904af77c0-metrics-client-ca\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.095043 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.094947    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ef08f252-a2f1-45ee-94a9-bb148e841f90-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:52.096467 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.096443    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef08f252-a2f1-45ee-94a9-bb148e841f90-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:52.096598 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.096541    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/872e9212-3c43-4ef7-94aa-b27904af77c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.097182 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.097160    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41e43ece-8336-4b4c-9355-feaa49cd2d2c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-ksvcr\" (UID: \"41e43ece-8336-4b4c-9355-feaa49cd2d2c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"
Sep 17 20:51:52.097264 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.097167    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/872e9212-3c43-4ef7-94aa-b27904af77c0-node-exporter-tls\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.097462 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.097441    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41e43ece-8336-4b4c-9355-feaa49cd2d2c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-ksvcr\" (UID: \"41e43ece-8336-4b4c-9355-feaa49cd2d2c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"
Sep 17 20:51:52.110585 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.110542    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxqf4\" (UniqueName: \"kubernetes.io/projected/872e9212-3c43-4ef7-94aa-b27904af77c0-kube-api-access-nxqf4\") pod \"node-exporter-dbcl5\" (UID: \"872e9212-3c43-4ef7-94aa-b27904af77c0\") " pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.110678 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.110661    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gknl\" (UniqueName: \"kubernetes.io/projected/ef08f252-a2f1-45ee-94a9-bb148e841f90-kube-api-access-4gknl\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:52.110926 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.110910    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcldx\" (UniqueName: \"kubernetes.io/projected/41e43ece-8336-4b4c-9355-feaa49cd2d2c-kube-api-access-hcldx\") pod \"openshift-state-metrics-86c5bc79d6-ksvcr\" (UID: \"41e43ece-8336-4b4c-9355-feaa49cd2d2c\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"
Sep 17 20:51:52.129707 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.129678    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"
Sep 17 20:51:52.160345 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.160312    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dbcl5"
Sep 17 20:51:52.172238 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:52.172203    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872e9212_3c43_4ef7_94aa_b27904af77c0.slice/crio-7f4de9625d3c2b6f4b05bfa4223d16f7352e59ec53763e19c50bc6cadc17de07 WatchSource:0}: Error finding container 7f4de9625d3c2b6f4b05bfa4223d16f7352e59ec53763e19c50bc6cadc17de07: Status 404 returned error can't find the container with id 7f4de9625d3c2b6f4b05bfa4223d16f7352e59ec53763e19c50bc6cadc17de07
Sep 17 20:51:52.179822 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.179149    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbcl5" event={"ID":"872e9212-3c43-4ef7-94aa-b27904af77c0","Type":"ContainerStarted","Data":"7f4de9625d3c2b6f4b05bfa4223d16f7352e59ec53763e19c50bc6cadc17de07"}
Sep 17 20:51:52.404433 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.404407    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr"]
Sep 17 20:51:52.406777 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:52.406750    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e43ece_8336_4b4c_9355_feaa49cd2d2c.slice/crio-74b505af1c7b0f28191f18239284e0833612455bbf6fb220094705bb7979493a WatchSource:0}: Error finding container 74b505af1c7b0f28191f18239284e0833612455bbf6fb220094705bb7979493a: Status 404 returned error can't find the container with id 74b505af1c7b0f28191f18239284e0833612455bbf6fb220094705bb7979493a
Sep 17 20:51:52.597376 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.597336    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef08f252-a2f1-45ee-94a9-bb148e841f90-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:52.599950 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.599928    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef08f252-a2f1-45ee-94a9-bb148e841f90-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-x4dbc\" (UID: \"ef08f252-a2f1-45ee-94a9-bb148e841f90\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:52.739415 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.739326    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"
Sep 17 20:51:52.960851 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.960681    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:51:52.964634 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.964608    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:52.967739 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.967715    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 20:51:52.968619 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.968034    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 20:51:52.968619 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.968088    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 20:51:52.968619 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.968088    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 20:51:52.968619 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.968312    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 20:51:52.968619 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.968339    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 20:51:52.968619 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.968400    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 20:51:52.968619 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.968422    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ff5hw\""
Sep 17 20:51:52.968619 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.968569    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 20:51:52.968619 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.968587    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 20:51:52.982835 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:52.982813    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:51:53.034300 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.034095    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-x4dbc"]
Sep 17 20:51:53.103195 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.103160    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/009566dd-7ba3-4549-8b14-8c62863f663d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.103380 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.103210    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/009566dd-7ba3-4549-8b14-8c62863f663d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.103380 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.103239    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-web-config\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.103380 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.103274    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/009566dd-7ba3-4549-8b14-8c62863f663d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.103380 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.103299    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.103380 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.103348    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.103688 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.103423    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/009566dd-7ba3-4549-8b14-8c62863f663d-config-out\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.103688 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.103452    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.103688 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.103487    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/009566dd-7ba3-4549-8b14-8c62863f663d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.103688 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.103512    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6px9\" (UniqueName: \"kubernetes.io/projected/009566dd-7ba3-4549-8b14-8c62863f663d-kube-api-access-l6px9\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.103688 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.103622    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-config-volume\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.103688 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.103663    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.103982 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.103700    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.189365 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.188985    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr" event={"ID":"41e43ece-8336-4b4c-9355-feaa49cd2d2c","Type":"ContainerStarted","Data":"88d211d03566faa36f5a5af20c0d1edd4ec2ee77c89fa2d9aa3e37ff2dc55932"}
Sep 17 20:51:53.189365 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.189055    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr" event={"ID":"41e43ece-8336-4b4c-9355-feaa49cd2d2c","Type":"ContainerStarted","Data":"18295ebb65ce2889b02e97f65b3b793a3772a245b2f8b6a2d323134c203056c9"}
Sep 17 20:51:53.189365 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.189083    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr" event={"ID":"41e43ece-8336-4b4c-9355-feaa49cd2d2c","Type":"ContainerStarted","Data":"74b505af1c7b0f28191f18239284e0833612455bbf6fb220094705bb7979493a"}
Sep 17 20:51:53.205832 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.204476    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-web-config\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.205832 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.204534    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/009566dd-7ba3-4549-8b14-8c62863f663d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.205832 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.204596    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.205832 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.204622    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.205832 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.204652    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/009566dd-7ba3-4549-8b14-8c62863f663d-config-out\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.205832 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.204677    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.205832 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.204715    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/009566dd-7ba3-4549-8b14-8c62863f663d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.205832 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.204739    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6px9\" (UniqueName: \"kubernetes.io/projected/009566dd-7ba3-4549-8b14-8c62863f663d-kube-api-access-l6px9\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.205832 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.204813    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-config-volume\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.205832 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.204841    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.205832 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.204888    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.205832 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.204952    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/009566dd-7ba3-4549-8b14-8c62863f663d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.205832 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.204975    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/009566dd-7ba3-4549-8b14-8c62863f663d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.205832 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.205722    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/009566dd-7ba3-4549-8b14-8c62863f663d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.206888 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.206791    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/009566dd-7ba3-4549-8b14-8c62863f663d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.207631 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.207599    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/009566dd-7ba3-4549-8b14-8c62863f663d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.210425 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.210403    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/009566dd-7ba3-4549-8b14-8c62863f663d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.210725 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.210682    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/009566dd-7ba3-4549-8b14-8c62863f663d-config-out\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.213273 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.213202    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.214253 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.214204    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.214366 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.214314    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.214877 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.214709    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-web-config\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.214877 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.214807    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-config-volume\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.214877 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.214844    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.215158 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.215138    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/009566dd-7ba3-4549-8b14-8c62863f663d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.220792 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.220770    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6px9\" (UniqueName: \"kubernetes.io/projected/009566dd-7ba3-4549-8b14-8c62863f663d-kube-api-access-l6px9\") pod \"alertmanager-main-0\" (UID: \"009566dd-7ba3-4549-8b14-8c62863f663d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.267654 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:53.267608    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef08f252_a2f1_45ee_94a9_bb148e841f90.slice/crio-670d92f8109a1d90c8513cf4926b8291dd7abd0136b202a60c88a48d8cb3e279 WatchSource:0}: Error finding container 670d92f8109a1d90c8513cf4926b8291dd7abd0136b202a60c88a48d8cb3e279: Status 404 returned error can't find the container with id 670d92f8109a1d90c8513cf4926b8291dd7abd0136b202a60c88a48d8cb3e279
Sep 17 20:51:53.277837 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.277812    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 20:51:53.617917 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:53.617889    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 20:51:53.622333 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:53.622305    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009566dd_7ba3_4549_8b14_8c62863f663d.slice/crio-4027c43cec52be27fd54b747257e8dbf8f649585af59b7bda9a2c82cb636f3dd WatchSource:0}: Error finding container 4027c43cec52be27fd54b747257e8dbf8f649585af59b7bda9a2c82cb636f3dd: Status 404 returned error can't find the container with id 4027c43cec52be27fd54b747257e8dbf8f649585af59b7bda9a2c82cb636f3dd
Sep 17 20:51:54.196939 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.196854    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr" event={"ID":"41e43ece-8336-4b4c-9355-feaa49cd2d2c","Type":"ContainerStarted","Data":"15776be8666aae95751917dab38a7a5b6cdce6fb8487a7b116047e4a12bf95c2"}
Sep 17 20:51:54.198804 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.198774    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc" event={"ID":"ef08f252-a2f1-45ee-94a9-bb148e841f90","Type":"ContainerStarted","Data":"670d92f8109a1d90c8513cf4926b8291dd7abd0136b202a60c88a48d8cb3e279"}
Sep 17 20:51:54.201017 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.200991    2571 generic.go:356] "Generic (PLEG): container finished" podID="872e9212-3c43-4ef7-94aa-b27904af77c0" containerID="dd8230784acea2df6f22563fb8421f1bddb65ebf9c976fe5c4bd9d5eebcc4c9e" exitCode=0
Sep 17 20:51:54.201126 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.201083    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbcl5" event={"ID":"872e9212-3c43-4ef7-94aa-b27904af77c0","Type":"ContainerDied","Data":"dd8230784acea2df6f22563fb8421f1bddb65ebf9c976fe5c4bd9d5eebcc4c9e"}
Sep 17 20:51:54.202854 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.202831    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"009566dd-7ba3-4549-8b14-8c62863f663d","Type":"ContainerStarted","Data":"4027c43cec52be27fd54b747257e8dbf8f649585af59b7bda9a2c82cb636f3dd"}
Sep 17 20:51:54.206045 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.206015    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl" podUID="acd16910-0f29-4e02-8cb9-d65894ee8d1a" containerName="registry" containerID="cri-o://dbdac0970f7056d4356606158303afdac804cfe2e669e46a1b4069847ecf3551" gracePeriod=30
Sep 17 20:51:54.216473 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.216414    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ksvcr" podStartSLOduration=2.167187458 podStartE2EDuration="3.21639517s" totalImagesPullingTime="1.049207712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:54.216207601 +0000 UTC m=+125.720685199" watchObservedRunningTime="2026-09-17 20:51:54.21639517 +0000 UTC m=+125.720872768"
Sep 17 20:51:54.648103 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.648079    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:54.722353 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.719595    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xwkw\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-kube-api-access-6xwkw\") pod \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") "
Sep 17 20:51:54.722353 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.719658    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acd16910-0f29-4e02-8cb9-d65894ee8d1a-trusted-ca\") pod \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") "
Sep 17 20:51:54.722353 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.719690    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-certificates\") pod \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") "
Sep 17 20:51:54.722353 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.719731    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acd16910-0f29-4e02-8cb9-d65894ee8d1a-installation-pull-secrets\") pod \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") "
Sep 17 20:51:54.722353 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.719762    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls\") pod \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") "
Sep 17 20:51:54.722353 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.719798    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acd16910-0f29-4e02-8cb9-d65894ee8d1a-ca-trust-extracted\") pod \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") "
Sep 17 20:51:54.722353 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.719833    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-bound-sa-token\") pod \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") "
Sep 17 20:51:54.722353 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.719874    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acd16910-0f29-4e02-8cb9-d65894ee8d1a-image-registry-private-configuration\") pod \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\" (UID: \"acd16910-0f29-4e02-8cb9-d65894ee8d1a\") "
Sep 17 20:51:54.722832 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.722705    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "acd16910-0f29-4e02-8cb9-d65894ee8d1a" (UID: "acd16910-0f29-4e02-8cb9-d65894ee8d1a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:51:54.722832 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.722714    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd16910-0f29-4e02-8cb9-d65894ee8d1a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "acd16910-0f29-4e02-8cb9-d65894ee8d1a" (UID: "acd16910-0f29-4e02-8cb9-d65894ee8d1a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:51:54.724593 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.723753    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-kube-api-access-6xwkw" (OuterVolumeSpecName: "kube-api-access-6xwkw") pod "acd16910-0f29-4e02-8cb9-d65894ee8d1a" (UID: "acd16910-0f29-4e02-8cb9-d65894ee8d1a"). InnerVolumeSpecName "kube-api-access-6xwkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:51:54.724593 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.724146    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "acd16910-0f29-4e02-8cb9-d65894ee8d1a" (UID: "acd16910-0f29-4e02-8cb9-d65894ee8d1a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:51:54.725666 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.725636    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd16910-0f29-4e02-8cb9-d65894ee8d1a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "acd16910-0f29-4e02-8cb9-d65894ee8d1a" (UID: "acd16910-0f29-4e02-8cb9-d65894ee8d1a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:51:54.726158 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.726137    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "acd16910-0f29-4e02-8cb9-d65894ee8d1a" (UID: "acd16910-0f29-4e02-8cb9-d65894ee8d1a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:51:54.726423 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.726313    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd16910-0f29-4e02-8cb9-d65894ee8d1a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "acd16910-0f29-4e02-8cb9-d65894ee8d1a" (UID: "acd16910-0f29-4e02-8cb9-d65894ee8d1a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:51:54.736439 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.736372    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd16910-0f29-4e02-8cb9-d65894ee8d1a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "acd16910-0f29-4e02-8cb9-d65894ee8d1a" (UID: "acd16910-0f29-4e02-8cb9-d65894ee8d1a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:51:54.821705 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.821632    2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acd16910-0f29-4e02-8cb9-d65894ee8d1a-ca-trust-extracted\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:51:54.821705 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.821666    2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-bound-sa-token\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:51:54.821705 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.821682    2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acd16910-0f29-4e02-8cb9-d65894ee8d1a-image-registry-private-configuration\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:51:54.821705 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.821697    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xwkw\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-kube-api-access-6xwkw\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:51:54.821705 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.821711    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acd16910-0f29-4e02-8cb9-d65894ee8d1a-trusted-ca\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:51:54.822080 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.821725    2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-certificates\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:51:54.822080 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.821738    2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acd16910-0f29-4e02-8cb9-d65894ee8d1a-installation-pull-secrets\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:51:54.822080 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:54.821751    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acd16910-0f29-4e02-8cb9-d65894ee8d1a-registry-tls\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:51:55.209498 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.209460    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc" event={"ID":"ef08f252-a2f1-45ee-94a9-bb148e841f90","Type":"ContainerStarted","Data":"c1d20d8ef616ffb502ac0bfad06646174685897d318e7693ebffd6306ab105af"}
Sep 17 20:51:55.209942 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.209568    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc" event={"ID":"ef08f252-a2f1-45ee-94a9-bb148e841f90","Type":"ContainerStarted","Data":"df9ad58bb67c6f286cc612fd2d8a221040925ab89c7f9559634adb304cd86c9d"}
Sep 17 20:51:55.209942 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.209591    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc" event={"ID":"ef08f252-a2f1-45ee-94a9-bb148e841f90","Type":"ContainerStarted","Data":"e133e35c1ab6acc9adf6f14330afcecf4891455bc7809d54b56d47d38895bc6e"}
Sep 17 20:51:55.212026 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.211995    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbcl5" event={"ID":"872e9212-3c43-4ef7-94aa-b27904af77c0","Type":"ContainerStarted","Data":"c1f698e57e96d1d64ed10b4783d6c92ec25efb7c1ca8764b84900a8ab2648de0"}
Sep 17 20:51:55.212138 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.212062    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dbcl5" event={"ID":"872e9212-3c43-4ef7-94aa-b27904af77c0","Type":"ContainerStarted","Data":"e2ea0869a6dab63f1f91b21ad6166b12b4805fdc6d781ddad54ce4a7e42a3071"}
Sep 17 20:51:55.213498 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.213473    2571 generic.go:356] "Generic (PLEG): container finished" podID="009566dd-7ba3-4549-8b14-8c62863f663d" containerID="35febf8014f43344767cc642dc439ca7a6e1c82c0423049534692d5c09861208" exitCode=0
Sep 17 20:51:55.213633 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.213572    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"009566dd-7ba3-4549-8b14-8c62863f663d","Type":"ContainerDied","Data":"35febf8014f43344767cc642dc439ca7a6e1c82c0423049534692d5c09861208"}
Sep 17 20:51:55.214847 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.214824    2571 generic.go:356] "Generic (PLEG): container finished" podID="acd16910-0f29-4e02-8cb9-d65894ee8d1a" containerID="dbdac0970f7056d4356606158303afdac804cfe2e669e46a1b4069847ecf3551" exitCode=0
Sep 17 20:51:55.214932 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.214883    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl"
Sep 17 20:51:55.214932 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.214909    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl" event={"ID":"acd16910-0f29-4e02-8cb9-d65894ee8d1a","Type":"ContainerDied","Data":"dbdac0970f7056d4356606158303afdac804cfe2e669e46a1b4069847ecf3551"}
Sep 17 20:51:55.215042 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.214962    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68747b9cb5-fnnrl" event={"ID":"acd16910-0f29-4e02-8cb9-d65894ee8d1a","Type":"ContainerDied","Data":"23349403e766554843f978917c6fc67d14a968ee8c0d6a7411ceef1575e4d381"}
Sep 17 20:51:55.215042 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.215004    2571 scope.go:117] "RemoveContainer" containerID="dbdac0970f7056d4356606158303afdac804cfe2e669e46a1b4069847ecf3551"
Sep 17 20:51:55.224718 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.224699    2571 scope.go:117] "RemoveContainer" containerID="dbdac0970f7056d4356606158303afdac804cfe2e669e46a1b4069847ecf3551"
Sep 17 20:51:55.224989 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:55.224969    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbdac0970f7056d4356606158303afdac804cfe2e669e46a1b4069847ecf3551\": container with ID starting with dbdac0970f7056d4356606158303afdac804cfe2e669e46a1b4069847ecf3551 not found: ID does not exist" containerID="dbdac0970f7056d4356606158303afdac804cfe2e669e46a1b4069847ecf3551"
Sep 17 20:51:55.225042 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.224997    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbdac0970f7056d4356606158303afdac804cfe2e669e46a1b4069847ecf3551"} err="failed to get container status \"dbdac0970f7056d4356606158303afdac804cfe2e669e46a1b4069847ecf3551\": rpc error: code = NotFound desc = could not find container \"dbdac0970f7056d4356606158303afdac804cfe2e669e46a1b4069847ecf3551\": container with ID starting with dbdac0970f7056d4356606158303afdac804cfe2e669e46a1b4069847ecf3551 not found: ID does not exist"
Sep 17 20:51:55.234736 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.234695    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-x4dbc" podStartSLOduration=3.133986322 podStartE2EDuration="4.234681134s" totalImagesPullingTime="1.100694812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:55.233604439 +0000 UTC m=+126.738082039" watchObservedRunningTime="2026-09-17 20:51:55.234681134 +0000 UTC m=+126.739158729"
Sep 17 20:51:55.279994 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.279963    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68747b9cb5-fnnrl"]
Sep 17 20:51:55.282798 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.282776    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68747b9cb5-fnnrl"]
Sep 17 20:51:55.300195 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:55.300147    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dbcl5" podStartSLOduration=3.08030038 podStartE2EDuration="4.30013125s" totalImagesPullingTime="1.138342211s" totalInitContainerRuntime="81.488659ms" isStatefulPod=true podCreationTimestamp="2026-09-17 20:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:55.29845526 +0000 UTC m=+126.802932869" watchObservedRunningTime="2026-09-17 20:51:55.30013125 +0000 UTC m=+126.804608846"
Sep 17 20:51:56.212035 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.211994    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-984fd59-brmlv"]
Sep 17 20:51:56.212699 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.212675    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acd16910-0f29-4e02-8cb9-d65894ee8d1a" containerName="registry"
Sep 17 20:51:56.212699 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.212699    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="acd16910-0f29-4e02-8cb9-d65894ee8d1a" containerName="registry"
Sep 17 20:51:56.212889 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.212779    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="acd16910-0f29-4e02-8cb9-d65894ee8d1a" containerName="registry"
Sep 17 20:51:56.215992 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.215969    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.220367 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.220341    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-f64zk\""
Sep 17 20:51:56.220714 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.220693    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 20:51:56.221778 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.221750    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 20:51:56.222039 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.221822    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2s10gug1qjlvr\""
Sep 17 20:51:56.222039 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.221884    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 20:51:56.222154 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.222112    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 20:51:56.234275 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.234245    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5ed028c-8213-427d-a77f-f4d78e7fd1da-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.234433 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.234375    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5ed028c-8213-427d-a77f-f4d78e7fd1da-secret-metrics-server-client-certs\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.234965 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.234676    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5ed028c-8213-427d-a77f-f4d78e7fd1da-audit-log\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.234965 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.234754    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5ed028c-8213-427d-a77f-f4d78e7fd1da-client-ca-bundle\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.234965 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.234818    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5ed028c-8213-427d-a77f-f4d78e7fd1da-secret-metrics-server-tls\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.234965 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.234859    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5ed028c-8213-427d-a77f-f4d78e7fd1da-metrics-server-audit-profiles\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.234965 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.234891    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlvc7\" (UniqueName: \"kubernetes.io/projected/d5ed028c-8213-427d-a77f-f4d78e7fd1da-kube-api-access-qlvc7\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.296402 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.296367    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-984fd59-brmlv"]
Sep 17 20:51:56.336200 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.336171    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5ed028c-8213-427d-a77f-f4d78e7fd1da-client-ca-bundle\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.336347 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.336208    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5ed028c-8213-427d-a77f-f4d78e7fd1da-secret-metrics-server-tls\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.336347 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.336232    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5ed028c-8213-427d-a77f-f4d78e7fd1da-metrics-server-audit-profiles\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.336347 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.336257    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlvc7\" (UniqueName: \"kubernetes.io/projected/d5ed028c-8213-427d-a77f-f4d78e7fd1da-kube-api-access-qlvc7\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.336347 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.336299    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5ed028c-8213-427d-a77f-f4d78e7fd1da-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.336566 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.336383    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5ed028c-8213-427d-a77f-f4d78e7fd1da-secret-metrics-server-client-certs\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.336566 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.336454    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5ed028c-8213-427d-a77f-f4d78e7fd1da-audit-log\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.336805 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.336779    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5ed028c-8213-427d-a77f-f4d78e7fd1da-audit-log\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.337573 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.337524    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5ed028c-8213-427d-a77f-f4d78e7fd1da-metrics-server-audit-profiles\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.338052 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.338024    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5ed028c-8213-427d-a77f-f4d78e7fd1da-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.339097 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.339070    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5ed028c-8213-427d-a77f-f4d78e7fd1da-secret-metrics-server-client-certs\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.339240 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.339223    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5ed028c-8213-427d-a77f-f4d78e7fd1da-secret-metrics-server-tls\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.339449 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.339427    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5ed028c-8213-427d-a77f-f4d78e7fd1da-client-ca-bundle\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.353642 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.353620    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlvc7\" (UniqueName: \"kubernetes.io/projected/d5ed028c-8213-427d-a77f-f4d78e7fd1da-kube-api-access-qlvc7\") pod \"metrics-server-984fd59-brmlv\" (UID: \"d5ed028c-8213-427d-a77f-f4d78e7fd1da\") " pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.529166 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.529083    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:51:56.649763 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.649733    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7498dcb95d-t6jpb"]
Sep 17 20:51:56.652903 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.652888    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.678714 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.678677    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acd16910-0f29-4e02-8cb9-d65894ee8d1a" path="/var/lib/kubelet/pods/acd16910-0f29-4e02-8cb9-d65894ee8d1a/volumes"
Sep 17 20:51:56.679052 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.679028    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb"]
Sep 17 20:51:56.682344 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.682319    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7498dcb95d-t6jpb"]
Sep 17 20:51:56.682465 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.682451    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb"
Sep 17 20:51:56.686045 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.686020    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 20:51:56.686168 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.686034    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8l2bs\""
Sep 17 20:51:56.698439 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.698412    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb"]
Sep 17 20:51:56.740137 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.740109    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74f4z\" (UniqueName: \"kubernetes.io/projected/a49ced7b-edb8-4aa6-a993-ee060eba26d8-kube-api-access-74f4z\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.740278 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.740161    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttk99\" (UniqueName: \"kubernetes.io/projected/34853159-04ae-4640-b9cf-804bcc9ab35f-kube-api-access-ttk99\") pod \"monitoring-plugin-5984459ff8-dt6xb\" (UID: \"34853159-04ae-4640-b9cf-804bcc9ab35f\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb"
Sep 17 20:51:56.740278 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.740185    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-trusted-ca-bundle\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.740278 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.740242    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-serving-cert\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.740278 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.740276    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-oauth-serving-cert\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.740504 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.740302    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-service-ca\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.740504 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.740327    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-config\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.740504 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.740376    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34853159-04ae-4640-b9cf-804bcc9ab35f-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dt6xb\" (UID: \"34853159-04ae-4640-b9cf-804bcc9ab35f\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb"
Sep 17 20:51:56.740504 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.740405    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-oauth-config\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.841744 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.841716    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-config\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.841914 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.841759    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34853159-04ae-4640-b9cf-804bcc9ab35f-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dt6xb\" (UID: \"34853159-04ae-4640-b9cf-804bcc9ab35f\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb"
Sep 17 20:51:56.841914 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.841790    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-oauth-config\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.841914 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.841813    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74f4z\" (UniqueName: \"kubernetes.io/projected/a49ced7b-edb8-4aa6-a993-ee060eba26d8-kube-api-access-74f4z\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.841914 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.841868    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttk99\" (UniqueName: \"kubernetes.io/projected/34853159-04ae-4640-b9cf-804bcc9ab35f-kube-api-access-ttk99\") pod \"monitoring-plugin-5984459ff8-dt6xb\" (UID: \"34853159-04ae-4640-b9cf-804bcc9ab35f\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb"
Sep 17 20:51:56.841914 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.841896    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-trusted-ca-bundle\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.842145 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.841937    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-serving-cert\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.842145 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.842001    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-oauth-serving-cert\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.842145 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.842052    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-service-ca\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.842833 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:56.842464    2571 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 20:51:56.842833 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:51:56.842539    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34853159-04ae-4640-b9cf-804bcc9ab35f-monitoring-plugin-cert podName:34853159-04ae-4640-b9cf-804bcc9ab35f nodeName:}" failed. No retries permitted until 2026-09-17 20:51:57.342519213 +0000 UTC m=+128.846996804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/34853159-04ae-4640-b9cf-804bcc9ab35f-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-dt6xb" (UID: "34853159-04ae-4640-b9cf-804bcc9ab35f") : secret "monitoring-plugin-cert" not found
Sep 17 20:51:56.842833 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.842812    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-service-ca\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.843247 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.843187    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-trusted-ca-bundle\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.843516 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.843496    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-oauth-serving-cert\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.844008 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.843988    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-config\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.845813 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.845766    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-serving-cert\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.846485 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.846450    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-oauth-config\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.859737 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.859687    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74f4z\" (UniqueName: \"kubernetes.io/projected/a49ced7b-edb8-4aa6-a993-ee060eba26d8-kube-api-access-74f4z\") pod \"console-7498dcb95d-t6jpb\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") " pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:56.865457 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.865432    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttk99\" (UniqueName: \"kubernetes.io/projected/34853159-04ae-4640-b9cf-804bcc9ab35f-kube-api-access-ttk99\") pod \"monitoring-plugin-5984459ff8-dt6xb\" (UID: \"34853159-04ae-4640-b9cf-804bcc9ab35f\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb"
Sep 17 20:51:56.936265 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.936233    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-984fd59-brmlv"]
Sep 17 20:51:56.942494 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:56.942465    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ed028c_8213_427d_a77f_f4d78e7fd1da.slice/crio-f388755c4ce597c18c67d64c727a1493bb72aa8e903fb486968c9b4ab033bbfe WatchSource:0}: Error finding container f388755c4ce597c18c67d64c727a1493bb72aa8e903fb486968c9b4ab033bbfe: Status 404 returned error can't find the container with id f388755c4ce597c18c67d64c727a1493bb72aa8e903fb486968c9b4ab033bbfe
Sep 17 20:51:56.962574 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:56.962537    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:51:57.228846 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:57.228811    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"009566dd-7ba3-4549-8b14-8c62863f663d","Type":"ContainerStarted","Data":"c2ecd7d05b9a2b8079f4cbb99c7468cbb40c3f59c06ea10df6e035de633ec230"}
Sep 17 20:51:57.229346 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:57.229318    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"009566dd-7ba3-4549-8b14-8c62863f663d","Type":"ContainerStarted","Data":"42f455629314b2f8afddad251577e566581286ef882cfdff3844c7516dca308d"}
Sep 17 20:51:57.229447 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:57.229366    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"009566dd-7ba3-4549-8b14-8c62863f663d","Type":"ContainerStarted","Data":"964962a5314b9b3613487a26088b93b748645a30398ca50e3a73eaad2559f3a6"}
Sep 17 20:51:57.229447 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:57.229410    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"009566dd-7ba3-4549-8b14-8c62863f663d","Type":"ContainerStarted","Data":"82c70270c1bfe72867f5e1fbbef825f83ea39cea3e37fa6ab68366c34aa30444"}
Sep 17 20:51:57.229589 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:57.229449    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"009566dd-7ba3-4549-8b14-8c62863f663d","Type":"ContainerStarted","Data":"264fa8b0632b1cb850b0705ae61b899488006437238903beddfb60beaf5cf864"}
Sep 17 20:51:57.230101 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:57.230084    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-984fd59-brmlv" event={"ID":"d5ed028c-8213-427d-a77f-f4d78e7fd1da","Type":"ContainerStarted","Data":"f388755c4ce597c18c67d64c727a1493bb72aa8e903fb486968c9b4ab033bbfe"}
Sep 17 20:51:57.244228 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:57.244205    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7498dcb95d-t6jpb"]
Sep 17 20:51:57.246355 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:57.246333    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49ced7b_edb8_4aa6_a993_ee060eba26d8.slice/crio-8780c41fae03fca21c2bf12390792140a96ff1f3352904e45995301393996a72 WatchSource:0}: Error finding container 8780c41fae03fca21c2bf12390792140a96ff1f3352904e45995301393996a72: Status 404 returned error can't find the container with id 8780c41fae03fca21c2bf12390792140a96ff1f3352904e45995301393996a72
Sep 17 20:51:57.345965 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:57.345933    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34853159-04ae-4640-b9cf-804bcc9ab35f-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dt6xb\" (UID: \"34853159-04ae-4640-b9cf-804bcc9ab35f\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb"
Sep 17 20:51:57.348642 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:57.348615    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34853159-04ae-4640-b9cf-804bcc9ab35f-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dt6xb\" (UID: \"34853159-04ae-4640-b9cf-804bcc9ab35f\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb"
Sep 17 20:51:57.593162 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:57.593129    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb"
Sep 17 20:51:57.875401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:57.875374    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb"]
Sep 17 20:51:57.888802 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:51:57.888769    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34853159_04ae_4640_b9cf_804bcc9ab35f.slice/crio-8eb255d7a2f7f0150f8c59f25b5035683636ce3b80b606e6d5942c075f02ce1e WatchSource:0}: Error finding container 8eb255d7a2f7f0150f8c59f25b5035683636ce3b80b606e6d5942c075f02ce1e: Status 404 returned error can't find the container with id 8eb255d7a2f7f0150f8c59f25b5035683636ce3b80b606e6d5942c075f02ce1e
Sep 17 20:51:58.236772 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:58.236679    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb" event={"ID":"34853159-04ae-4640-b9cf-804bcc9ab35f","Type":"ContainerStarted","Data":"8eb255d7a2f7f0150f8c59f25b5035683636ce3b80b606e6d5942c075f02ce1e"}
Sep 17 20:51:58.238679 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:58.238647    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7498dcb95d-t6jpb" event={"ID":"a49ced7b-edb8-4aa6-a993-ee060eba26d8","Type":"ContainerStarted","Data":"5014249df1ee9f7764480746fca460583c1379dd9b8b13ab3de3d65a55860b69"}
Sep 17 20:51:58.238822 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:58.238701    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7498dcb95d-t6jpb" event={"ID":"a49ced7b-edb8-4aa6-a993-ee060eba26d8","Type":"ContainerStarted","Data":"8780c41fae03fca21c2bf12390792140a96ff1f3352904e45995301393996a72"}
Sep 17 20:51:58.243135 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:58.243113    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"009566dd-7ba3-4549-8b14-8c62863f663d","Type":"ContainerStarted","Data":"c01fa91c1fe8936779ece0b6283ec90bb82186f789b561750ed74ec5e8208925"}
Sep 17 20:51:58.275270 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:58.275209    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7498dcb95d-t6jpb" podStartSLOduration=2.275186724 podStartE2EDuration="2.275186724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:58.273574628 +0000 UTC m=+129.778052225" watchObservedRunningTime="2026-09-17 20:51:58.275186724 +0000 UTC m=+129.779664321"
Sep 17 20:51:58.307841 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:58.307790    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.923157669 podStartE2EDuration="6.30777041s" totalImagesPullingTime="3.359149923s" totalInitContainerRuntime="25.462818ms" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:58.306734504 +0000 UTC m=+129.811212101" watchObservedRunningTime="2026-09-17 20:51:58.30777041 +0000 UTC m=+129.812248006"
Sep 17 20:51:59.249779 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:59.249741    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-984fd59-brmlv" event={"ID":"d5ed028c-8213-427d-a77f-f4d78e7fd1da","Type":"ContainerStarted","Data":"bce012274896c711301a52ccc585b00555399c454db1616d2d010893c5405d23"}
Sep 17 20:51:59.276127 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:51:59.276080    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-984fd59-brmlv" podStartSLOduration=1.85946679 podStartE2EDuration="3.276064071s" totalImagesPullingTime="1.416597281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:59.275385156 +0000 UTC m=+130.779862750" watchObservedRunningTime="2026-09-17 20:51:59.276064071 +0000 UTC m=+130.780541667"
Sep 17 20:52:00.256272 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:00.256238    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb" event={"ID":"34853159-04ae-4640-b9cf-804bcc9ab35f","Type":"ContainerStarted","Data":"085f083ad4a466406a1ea1b7f32ec1e75008f1652e2e9040f01542f7b0f4be9a"}
Sep 17 20:52:00.256739 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:00.256576    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb"
Sep 17 20:52:00.260850 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:00.260821    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb"
Sep 17 20:52:00.277601 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:00.277530    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dt6xb" podStartSLOduration=2.9729024859999997 podStartE2EDuration="4.277513287s" totalImagesPullingTime="1.304610801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:00.276633114 +0000 UTC m=+131.781110720" watchObservedRunningTime="2026-09-17 20:52:00.277513287 +0000 UTC m=+131.781990883"
Sep 17 20:52:00.379317 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:00.379277    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs\") pod \"network-metrics-daemon-6742m\" (UID: \"d2215eda-bca2-49dd-9419-90f9736fdf41\") " pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:52:00.381904 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:00.381885    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2215eda-bca2-49dd-9419-90f9736fdf41-metrics-certs\") pod \"network-metrics-daemon-6742m\" (UID: \"d2215eda-bca2-49dd-9419-90f9736fdf41\") " pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:52:00.487830 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:00.487798    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9vrtf\""
Sep 17 20:52:00.496040 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:00.496009    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6742m"
Sep 17 20:52:00.777859 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:00.777834    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6742m"]
Sep 17 20:52:00.779883 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:52:00.779850    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2215eda_bca2_49dd_9419_90f9736fdf41.slice/crio-11389c60a5c7d8cf5e386837799b431ee75ae4264a839004a5e066774e6f5012 WatchSource:0}: Error finding container 11389c60a5c7d8cf5e386837799b431ee75ae4264a839004a5e066774e6f5012: Status 404 returned error can't find the container with id 11389c60a5c7d8cf5e386837799b431ee75ae4264a839004a5e066774e6f5012
Sep 17 20:52:01.264312 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.264272    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6742m" event={"ID":"d2215eda-bca2-49dd-9419-90f9736fdf41","Type":"ContainerStarted","Data":"11389c60a5c7d8cf5e386837799b431ee75ae4264a839004a5e066774e6f5012"}
Sep 17 20:52:01.797375 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.797345    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7498dcb95d-t6jpb"]
Sep 17 20:52:01.835503 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.835472    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-675969869c-ltvsw"]
Sep 17 20:52:01.839117 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.839091    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.849056 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.849028    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-675969869c-ltvsw"]
Sep 17 20:52:01.893655 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.893537    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5153c3b0-7122-4afc-8908-a44370ded276-console-serving-cert\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.893655 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.893606    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-oauth-serving-cert\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.893835 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.893662    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-service-ca\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.893835 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.893718    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5153c3b0-7122-4afc-8908-a44370ded276-console-oauth-config\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.893835 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.893747    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-trusted-ca-bundle\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.893835 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.893769    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-console-config\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.893835 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.893806    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjfdv\" (UniqueName: \"kubernetes.io/projected/5153c3b0-7122-4afc-8908-a44370ded276-kube-api-access-bjfdv\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.995175 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.995133    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjfdv\" (UniqueName: \"kubernetes.io/projected/5153c3b0-7122-4afc-8908-a44370ded276-kube-api-access-bjfdv\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.995367 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.995188    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5153c3b0-7122-4afc-8908-a44370ded276-console-serving-cert\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.995367 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.995205    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-oauth-serving-cert\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.995367 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.995239    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-service-ca\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.995367 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.995279    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5153c3b0-7122-4afc-8908-a44370ded276-console-oauth-config\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.995367 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.995319    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-trusted-ca-bundle\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.995367 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.995348    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-console-config\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.996235 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.996194    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-oauth-serving-cert\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.996235 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.996194    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-console-config\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.996390 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.996252    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-service-ca\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.996390 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.996292    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-trusted-ca-bundle\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.997949 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.997915    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5153c3b0-7122-4afc-8908-a44370ded276-console-serving-cert\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:01.998102 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:01.998082    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5153c3b0-7122-4afc-8908-a44370ded276-console-oauth-config\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:02.004098 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:02.004074    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjfdv\" (UniqueName: \"kubernetes.io/projected/5153c3b0-7122-4afc-8908-a44370ded276-kube-api-access-bjfdv\") pod \"console-675969869c-ltvsw\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") " pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:02.149063 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:02.148978    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:02.270817 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:02.270782    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6742m" event={"ID":"d2215eda-bca2-49dd-9419-90f9736fdf41","Type":"ContainerStarted","Data":"8c607f8962aa2b660176c98f78962a9412b9bca76cfbbfaaf8359453c8f7e2d5"}
Sep 17 20:52:02.271250 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:02.270834    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6742m" event={"ID":"d2215eda-bca2-49dd-9419-90f9736fdf41","Type":"ContainerStarted","Data":"7897346adafe0c9fab470fd1d692ed2d2d7824483dbf3920d3d8335785217819"}
Sep 17 20:52:02.292193 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:02.292139    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6742m" podStartSLOduration=131.445480666 podStartE2EDuration="2m12.292116979s" totalImagesPullingTime="846.636313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:02.289392006 +0000 UTC m=+133.793869614" watchObservedRunningTime="2026-09-17 20:52:02.292116979 +0000 UTC m=+133.796594575"
Sep 17 20:52:02.423789 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:02.423763    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-675969869c-ltvsw"]
Sep 17 20:52:02.426461 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:52:02.426436    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5153c3b0_7122_4afc_8908_a44370ded276.slice/crio-b6f2b87c500619be81d7df0c06f981fb707ebce3fbf01ecab2c888251f55abcc WatchSource:0}: Error finding container b6f2b87c500619be81d7df0c06f981fb707ebce3fbf01ecab2c888251f55abcc: Status 404 returned error can't find the container with id b6f2b87c500619be81d7df0c06f981fb707ebce3fbf01ecab2c888251f55abcc
Sep 17 20:52:03.276777 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:03.276746    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-675969869c-ltvsw" event={"ID":"5153c3b0-7122-4afc-8908-a44370ded276","Type":"ContainerStarted","Data":"a52aa575e2a7bb771fbf12a9a82f9001b110a923b88803762b3ef3d8f6260bb8"}
Sep 17 20:52:03.277245 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:03.276792    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-675969869c-ltvsw" event={"ID":"5153c3b0-7122-4afc-8908-a44370ded276","Type":"ContainerStarted","Data":"b6f2b87c500619be81d7df0c06f981fb707ebce3fbf01ecab2c888251f55abcc"}
Sep 17 20:52:06.962922 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:06.962890    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:52:11.314842 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:11.314805    2571 generic.go:356] "Generic (PLEG): container finished" podID="374b4cca-e914-4b51-a833-32dcbcd655f2" containerID="565fbfe56f92e2060115243f90330feda07569947b27d1424e9a64a22adba167" exitCode=0
Sep 17 20:52:11.315248 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:11.314875    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5" event={"ID":"374b4cca-e914-4b51-a833-32dcbcd655f2","Type":"ContainerDied","Data":"565fbfe56f92e2060115243f90330feda07569947b27d1424e9a64a22adba167"}
Sep 17 20:52:11.315313 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:11.315295    2571 scope.go:117] "RemoveContainer" containerID="565fbfe56f92e2060115243f90330feda07569947b27d1424e9a64a22adba167"
Sep 17 20:52:11.316523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:11.316500    2571 generic.go:356] "Generic (PLEG): container finished" podID="ad635b09-43a6-4d29-b7c0-792f8327751c" containerID="d533bed4a374c9af9f04c758184bc693b8f1df2640ff3f4a04ad582ede38f45f" exitCode=0
Sep 17 20:52:11.316645 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:11.316543    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5" event={"ID":"ad635b09-43a6-4d29-b7c0-792f8327751c","Type":"ContainerDied","Data":"d533bed4a374c9af9f04c758184bc693b8f1df2640ff3f4a04ad582ede38f45f"}
Sep 17 20:52:11.316897 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:11.316882    2571 scope.go:117] "RemoveContainer" containerID="d533bed4a374c9af9f04c758184bc693b8f1df2640ff3f4a04ad582ede38f45f"
Sep 17 20:52:11.333073 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:11.333025    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-675969869c-ltvsw" podStartSLOduration=10.333008142 podStartE2EDuration="10.333008142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:03.311976582 +0000 UTC m=+134.816454178" watchObservedRunningTime="2026-09-17 20:52:11.333008142 +0000 UTC m=+142.837485740"
Sep 17 20:52:12.149137 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:12.149105    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:12.149327 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:12.149167    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:12.154042 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:12.154014    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:12.322625 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:12.322590    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fczn5" event={"ID":"ad635b09-43a6-4d29-b7c0-792f8327751c","Type":"ContainerStarted","Data":"54a3f31b2c9d7c6f0060939042cd120760af9603b2851dfbd4dc373c2b6c739c"}
Sep 17 20:52:12.324767 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:12.324739    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xbdx5" event={"ID":"374b4cca-e914-4b51-a833-32dcbcd655f2","Type":"ContainerStarted","Data":"29c44b94a3d630cd1bdf1431f0433bbb4acc2490d19a5dadaf1e8124fd92041b"}
Sep 17 20:52:12.329213 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:12.329191    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:52:12.410505 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:12.410423    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c5cdb566b-p7xlh"]
Sep 17 20:52:14.247089 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:14.247048    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bb8bfdffd-24zth" podUID="9e7c0e30-282e-41d9-90bc-39f26a48444a" containerName="console" containerID="cri-o://9dcf1151a4358f0c53b9c18702953cbe1f779446f75f6ca2e81e6f2c57d3b418" gracePeriod=15
Sep 17 20:52:14.990947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:14.990924    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:52:15.112645 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.112536    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-serving-cert\") pod \"9e7c0e30-282e-41d9-90bc-39f26a48444a\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") "
Sep 17 20:52:15.112645 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.112593    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-oauth-config\") pod \"9e7c0e30-282e-41d9-90bc-39f26a48444a\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") "
Sep 17 20:52:15.112645 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.112642    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-service-ca\") pod \"9e7c0e30-282e-41d9-90bc-39f26a48444a\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") "
Sep 17 20:52:15.112900 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.112660    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-oauth-serving-cert\") pod \"9e7c0e30-282e-41d9-90bc-39f26a48444a\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") "
Sep 17 20:52:15.112900 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.112693    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfspv\" (UniqueName: \"kubernetes.io/projected/9e7c0e30-282e-41d9-90bc-39f26a48444a-kube-api-access-qfspv\") pod \"9e7c0e30-282e-41d9-90bc-39f26a48444a\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") "
Sep 17 20:52:15.112900 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.112723    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-config\") pod \"9e7c0e30-282e-41d9-90bc-39f26a48444a\" (UID: \"9e7c0e30-282e-41d9-90bc-39f26a48444a\") "
Sep 17 20:52:15.113121 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.113088    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-service-ca" (OuterVolumeSpecName: "service-ca") pod "9e7c0e30-282e-41d9-90bc-39f26a48444a" (UID: "9e7c0e30-282e-41d9-90bc-39f26a48444a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:15.113121 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.113107    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9e7c0e30-282e-41d9-90bc-39f26a48444a" (UID: "9e7c0e30-282e-41d9-90bc-39f26a48444a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:15.113276 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.113131    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-config" (OuterVolumeSpecName: "console-config") pod "9e7c0e30-282e-41d9-90bc-39f26a48444a" (UID: "9e7c0e30-282e-41d9-90bc-39f26a48444a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:15.115132 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.115105    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9e7c0e30-282e-41d9-90bc-39f26a48444a" (UID: "9e7c0e30-282e-41d9-90bc-39f26a48444a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:15.115132 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.115127    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9e7c0e30-282e-41d9-90bc-39f26a48444a" (UID: "9e7c0e30-282e-41d9-90bc-39f26a48444a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:15.115262 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.115220    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7c0e30-282e-41d9-90bc-39f26a48444a-kube-api-access-qfspv" (OuterVolumeSpecName: "kube-api-access-qfspv") pod "9e7c0e30-282e-41d9-90bc-39f26a48444a" (UID: "9e7c0e30-282e-41d9-90bc-39f26a48444a"). InnerVolumeSpecName "kube-api-access-qfspv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:15.213711 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.213655    2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-serving-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:15.213711 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.213701    2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-oauth-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:15.213711 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.213711    2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-service-ca\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:15.213711 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.213721    2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-oauth-serving-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:15.213711 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.213729    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfspv\" (UniqueName: \"kubernetes.io/projected/9e7c0e30-282e-41d9-90bc-39f26a48444a-kube-api-access-qfspv\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:15.213998 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.213738    2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9e7c0e30-282e-41d9-90bc-39f26a48444a-console-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:15.340191 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.340152    2571 generic.go:356] "Generic (PLEG): container finished" podID="9e7c0e30-282e-41d9-90bc-39f26a48444a" containerID="9dcf1151a4358f0c53b9c18702953cbe1f779446f75f6ca2e81e6f2c57d3b418" exitCode=0
Sep 17 20:52:15.340646 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.340201    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bb8bfdffd-24zth"
Sep 17 20:52:15.340646 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.340246    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bb8bfdffd-24zth" event={"ID":"9e7c0e30-282e-41d9-90bc-39f26a48444a","Type":"ContainerDied","Data":"9dcf1151a4358f0c53b9c18702953cbe1f779446f75f6ca2e81e6f2c57d3b418"}
Sep 17 20:52:15.340646 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.340305    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bb8bfdffd-24zth" event={"ID":"9e7c0e30-282e-41d9-90bc-39f26a48444a","Type":"ContainerDied","Data":"928922813ee2fab85316ed59cf79323e8be8a7f30c508922d6958bfc82aca673"}
Sep 17 20:52:15.340646 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.340337    2571 scope.go:117] "RemoveContainer" containerID="9dcf1151a4358f0c53b9c18702953cbe1f779446f75f6ca2e81e6f2c57d3b418"
Sep 17 20:52:15.354771 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.354751    2571 scope.go:117] "RemoveContainer" containerID="9dcf1151a4358f0c53b9c18702953cbe1f779446f75f6ca2e81e6f2c57d3b418"
Sep 17 20:52:15.355116 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:52:15.355094    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dcf1151a4358f0c53b9c18702953cbe1f779446f75f6ca2e81e6f2c57d3b418\": container with ID starting with 9dcf1151a4358f0c53b9c18702953cbe1f779446f75f6ca2e81e6f2c57d3b418 not found: ID does not exist" containerID="9dcf1151a4358f0c53b9c18702953cbe1f779446f75f6ca2e81e6f2c57d3b418"
Sep 17 20:52:15.355178 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.355124    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dcf1151a4358f0c53b9c18702953cbe1f779446f75f6ca2e81e6f2c57d3b418"} err="failed to get container status \"9dcf1151a4358f0c53b9c18702953cbe1f779446f75f6ca2e81e6f2c57d3b418\": rpc error: code = NotFound desc = could not find container \"9dcf1151a4358f0c53b9c18702953cbe1f779446f75f6ca2e81e6f2c57d3b418\": container with ID starting with 9dcf1151a4358f0c53b9c18702953cbe1f779446f75f6ca2e81e6f2c57d3b418 not found: ID does not exist"
Sep 17 20:52:15.370213 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.370144    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bb8bfdffd-24zth"]
Sep 17 20:52:15.376035 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:15.376009    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bb8bfdffd-24zth"]
Sep 17 20:52:16.529634 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:16.529599    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:52:16.530009 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:16.529690    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:52:16.669609 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:16.669579    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e7c0e30-282e-41d9-90bc-39f26a48444a" path="/var/lib/kubelet/pods/9e7c0e30-282e-41d9-90bc-39f26a48444a/volumes"
Sep 17 20:52:26.389439 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:26.389403    2571 generic.go:356] "Generic (PLEG): container finished" podID="8bf9e322-db1c-4373-986b-d4d95d2dad3c" containerID="9bb01eee99ff051874df56d20a52fd7faa46834f37f8507d3968ad2ae3684f8e" exitCode=0
Sep 17 20:52:26.389979 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:26.389481    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw" event={"ID":"8bf9e322-db1c-4373-986b-d4d95d2dad3c","Type":"ContainerDied","Data":"9bb01eee99ff051874df56d20a52fd7faa46834f37f8507d3968ad2ae3684f8e"}
Sep 17 20:52:26.390054 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:26.389983    2571 scope.go:117] "RemoveContainer" containerID="9bb01eee99ff051874df56d20a52fd7faa46834f37f8507d3968ad2ae3684f8e"
Sep 17 20:52:26.818597 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:26.818537    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7498dcb95d-t6jpb" podUID="a49ced7b-edb8-4aa6-a993-ee060eba26d8" containerName="console" containerID="cri-o://5014249df1ee9f7764480746fca460583c1379dd9b8b13ab3de3d65a55860b69" gracePeriod=15
Sep 17 20:52:27.069275 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.069215    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:52:27.225568 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.225521    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-service-ca\") pod \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") "
Sep 17 20:52:27.225764 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.225615    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-config\") pod \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") "
Sep 17 20:52:27.225764 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.225644    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-trusted-ca-bundle\") pod \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") "
Sep 17 20:52:27.225764 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.225734    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74f4z\" (UniqueName: \"kubernetes.io/projected/a49ced7b-edb8-4aa6-a993-ee060eba26d8-kube-api-access-74f4z\") pod \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") "
Sep 17 20:52:27.225764 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.225761    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-serving-cert\") pod \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") "
Sep 17 20:52:27.225969 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.225785    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-oauth-serving-cert\") pod \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") "
Sep 17 20:52:27.225969 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.225809    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-oauth-config\") pod \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\" (UID: \"a49ced7b-edb8-4aa6-a993-ee060eba26d8\") "
Sep 17 20:52:27.226056 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.225962    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-service-ca" (OuterVolumeSpecName: "service-ca") pod "a49ced7b-edb8-4aa6-a993-ee060eba26d8" (UID: "a49ced7b-edb8-4aa6-a993-ee060eba26d8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:27.226110 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.226077    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-config" (OuterVolumeSpecName: "console-config") pod "a49ced7b-edb8-4aa6-a993-ee060eba26d8" (UID: "a49ced7b-edb8-4aa6-a993-ee060eba26d8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:27.226174 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.226128    2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-service-ca\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:27.226174 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.226121    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a49ced7b-edb8-4aa6-a993-ee060eba26d8" (UID: "a49ced7b-edb8-4aa6-a993-ee060eba26d8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:27.226174 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.226150    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a49ced7b-edb8-4aa6-a993-ee060eba26d8" (UID: "a49ced7b-edb8-4aa6-a993-ee060eba26d8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:27.228204 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.228169    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a49ced7b-edb8-4aa6-a993-ee060eba26d8" (UID: "a49ced7b-edb8-4aa6-a993-ee060eba26d8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:27.228204 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.228175    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49ced7b-edb8-4aa6-a993-ee060eba26d8-kube-api-access-74f4z" (OuterVolumeSpecName: "kube-api-access-74f4z") pod "a49ced7b-edb8-4aa6-a993-ee060eba26d8" (UID: "a49ced7b-edb8-4aa6-a993-ee060eba26d8"). InnerVolumeSpecName "kube-api-access-74f4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:27.228354 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.228309    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a49ced7b-edb8-4aa6-a993-ee060eba26d8" (UID: "a49ced7b-edb8-4aa6-a993-ee060eba26d8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:27.327071 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.326989    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74f4z\" (UniqueName: \"kubernetes.io/projected/a49ced7b-edb8-4aa6-a993-ee060eba26d8-kube-api-access-74f4z\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:27.327071 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.327017    2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-serving-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:27.327071 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.327025    2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-oauth-serving-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:27.327071 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.327033    2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-oauth-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:27.327071 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.327041    2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-console-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:27.327071 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.327049    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a49ced7b-edb8-4aa6-a993-ee060eba26d8-trusted-ca-bundle\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:27.395617 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.395583    2571 generic.go:356] "Generic (PLEG): container finished" podID="a49ced7b-edb8-4aa6-a993-ee060eba26d8" containerID="5014249df1ee9f7764480746fca460583c1379dd9b8b13ab3de3d65a55860b69" exitCode=0
Sep 17 20:52:27.396046 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.395630    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7498dcb95d-t6jpb"
Sep 17 20:52:27.396046 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.395642    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7498dcb95d-t6jpb" event={"ID":"a49ced7b-edb8-4aa6-a993-ee060eba26d8","Type":"ContainerDied","Data":"5014249df1ee9f7764480746fca460583c1379dd9b8b13ab3de3d65a55860b69"}
Sep 17 20:52:27.396046 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.395698    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7498dcb95d-t6jpb" event={"ID":"a49ced7b-edb8-4aa6-a993-ee060eba26d8","Type":"ContainerDied","Data":"8780c41fae03fca21c2bf12390792140a96ff1f3352904e45995301393996a72"}
Sep 17 20:52:27.396046 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.395726    2571 scope.go:117] "RemoveContainer" containerID="5014249df1ee9f7764480746fca460583c1379dd9b8b13ab3de3d65a55860b69"
Sep 17 20:52:27.397882 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.397854    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-4rvqw" event={"ID":"8bf9e322-db1c-4373-986b-d4d95d2dad3c","Type":"ContainerStarted","Data":"9dc8d6a5d2d490e70ba865ad977e618c2466a42546dd225e4456be9de0d3efc6"}
Sep 17 20:52:27.405416 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.405397    2571 scope.go:117] "RemoveContainer" containerID="5014249df1ee9f7764480746fca460583c1379dd9b8b13ab3de3d65a55860b69"
Sep 17 20:52:27.405725 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:52:27.405704    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5014249df1ee9f7764480746fca460583c1379dd9b8b13ab3de3d65a55860b69\": container with ID starting with 5014249df1ee9f7764480746fca460583c1379dd9b8b13ab3de3d65a55860b69 not found: ID does not exist" containerID="5014249df1ee9f7764480746fca460583c1379dd9b8b13ab3de3d65a55860b69"
Sep 17 20:52:27.405775 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.405733    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5014249df1ee9f7764480746fca460583c1379dd9b8b13ab3de3d65a55860b69"} err="failed to get container status \"5014249df1ee9f7764480746fca460583c1379dd9b8b13ab3de3d65a55860b69\": rpc error: code = NotFound desc = could not find container \"5014249df1ee9f7764480746fca460583c1379dd9b8b13ab3de3d65a55860b69\": container with ID starting with 5014249df1ee9f7764480746fca460583c1379dd9b8b13ab3de3d65a55860b69 not found: ID does not exist"
Sep 17 20:52:27.435980 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.435938    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7498dcb95d-t6jpb"]
Sep 17 20:52:27.445279 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:27.445247    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7498dcb95d-t6jpb"]
Sep 17 20:52:28.669092 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:28.669057    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a49ced7b-edb8-4aa6-a993-ee060eba26d8" path="/var/lib/kubelet/pods/a49ced7b-edb8-4aa6-a993-ee060eba26d8/volumes"
Sep 17 20:52:36.535028 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:36.534951    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:52:36.538952 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:36.538924    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-984fd59-brmlv"
Sep 17 20:52:37.437020 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.436949    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c5cdb566b-p7xlh" podUID="bcab22c0-5a7b-45db-b464-657a40d83e0e" containerName="console" containerID="cri-o://df89be0a78643f640b542c84cc418cf3960ed1d0c701f722a8e249b199c447db" gracePeriod=15
Sep 17 20:52:37.681614 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.681590    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:52:37.712618 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.712522    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp76z\" (UniqueName: \"kubernetes.io/projected/bcab22c0-5a7b-45db-b464-657a40d83e0e-kube-api-access-tp76z\") pod \"bcab22c0-5a7b-45db-b464-657a40d83e0e\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") "
Sep 17 20:52:37.712618 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.712610    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-oauth-config\") pod \"bcab22c0-5a7b-45db-b464-657a40d83e0e\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") "
Sep 17 20:52:37.712836 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.712640    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-serving-cert\") pod \"bcab22c0-5a7b-45db-b464-657a40d83e0e\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") "
Sep 17 20:52:37.712836 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.712687    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-service-ca\") pod \"bcab22c0-5a7b-45db-b464-657a40d83e0e\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") "
Sep 17 20:52:37.712836 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.712724    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-trusted-ca-bundle\") pod \"bcab22c0-5a7b-45db-b464-657a40d83e0e\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") "
Sep 17 20:52:37.712836 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.712788    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-config\") pod \"bcab22c0-5a7b-45db-b464-657a40d83e0e\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") "
Sep 17 20:52:37.712836 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.712817    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-oauth-serving-cert\") pod \"bcab22c0-5a7b-45db-b464-657a40d83e0e\" (UID: \"bcab22c0-5a7b-45db-b464-657a40d83e0e\") "
Sep 17 20:52:37.713245 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.713210    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-service-ca" (OuterVolumeSpecName: "service-ca") pod "bcab22c0-5a7b-45db-b464-657a40d83e0e" (UID: "bcab22c0-5a7b-45db-b464-657a40d83e0e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:37.713418 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.713297    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bcab22c0-5a7b-45db-b464-657a40d83e0e" (UID: "bcab22c0-5a7b-45db-b464-657a40d83e0e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:37.713418 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.713355    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bcab22c0-5a7b-45db-b464-657a40d83e0e" (UID: "bcab22c0-5a7b-45db-b464-657a40d83e0e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:37.713790 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.713753    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-config" (OuterVolumeSpecName: "console-config") pod "bcab22c0-5a7b-45db-b464-657a40d83e0e" (UID: "bcab22c0-5a7b-45db-b464-657a40d83e0e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:37.715050 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.715026    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bcab22c0-5a7b-45db-b464-657a40d83e0e" (UID: "bcab22c0-5a7b-45db-b464-657a40d83e0e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:37.715253 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.715225    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcab22c0-5a7b-45db-b464-657a40d83e0e-kube-api-access-tp76z" (OuterVolumeSpecName: "kube-api-access-tp76z") pod "bcab22c0-5a7b-45db-b464-657a40d83e0e" (UID: "bcab22c0-5a7b-45db-b464-657a40d83e0e"). InnerVolumeSpecName "kube-api-access-tp76z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:37.715915 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.715881    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bcab22c0-5a7b-45db-b464-657a40d83e0e" (UID: "bcab22c0-5a7b-45db-b464-657a40d83e0e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:37.813570 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.813527    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp76z\" (UniqueName: \"kubernetes.io/projected/bcab22c0-5a7b-45db-b464-657a40d83e0e-kube-api-access-tp76z\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:37.813570 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.813571    2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-oauth-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:37.813570 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.813580    2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-serving-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:37.813810 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.813589    2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-service-ca\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:37.813810 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.813597    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-trusted-ca-bundle\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:37.813810 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.813604    2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-console-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:37.813810 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:37.813612    2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bcab22c0-5a7b-45db-b464-657a40d83e0e-oauth-serving-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:52:38.455720 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:38.451579    2571 generic.go:356] "Generic (PLEG): container finished" podID="bcab22c0-5a7b-45db-b464-657a40d83e0e" containerID="df89be0a78643f640b542c84cc418cf3960ed1d0c701f722a8e249b199c447db" exitCode=0
Sep 17 20:52:38.455720 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:38.451673    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5cdb566b-p7xlh" event={"ID":"bcab22c0-5a7b-45db-b464-657a40d83e0e","Type":"ContainerDied","Data":"df89be0a78643f640b542c84cc418cf3960ed1d0c701f722a8e249b199c447db"}
Sep 17 20:52:38.455720 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:38.451724    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5cdb566b-p7xlh" event={"ID":"bcab22c0-5a7b-45db-b464-657a40d83e0e","Type":"ContainerDied","Data":"995aeae46fe15a90fa4e587f64e1efb0da973519cee33653bddb3778ca041902"}
Sep 17 20:52:38.455720 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:38.451756    2571 scope.go:117] "RemoveContainer" containerID="df89be0a78643f640b542c84cc418cf3960ed1d0c701f722a8e249b199c447db"
Sep 17 20:52:38.455720 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:38.451916    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5cdb566b-p7xlh"
Sep 17 20:52:38.465631 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:38.465571    2571 scope.go:117] "RemoveContainer" containerID="df89be0a78643f640b542c84cc418cf3960ed1d0c701f722a8e249b199c447db"
Sep 17 20:52:38.465952 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:52:38.465932    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df89be0a78643f640b542c84cc418cf3960ed1d0c701f722a8e249b199c447db\": container with ID starting with df89be0a78643f640b542c84cc418cf3960ed1d0c701f722a8e249b199c447db not found: ID does not exist" containerID="df89be0a78643f640b542c84cc418cf3960ed1d0c701f722a8e249b199c447db"
Sep 17 20:52:38.465996 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:38.465963    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df89be0a78643f640b542c84cc418cf3960ed1d0c701f722a8e249b199c447db"} err="failed to get container status \"df89be0a78643f640b542c84cc418cf3960ed1d0c701f722a8e249b199c447db\": rpc error: code = NotFound desc = could not find container \"df89be0a78643f640b542c84cc418cf3960ed1d0c701f722a8e249b199c447db\": container with ID starting with df89be0a78643f640b542c84cc418cf3960ed1d0c701f722a8e249b199c447db not found: ID does not exist"
Sep 17 20:52:38.477953 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:38.477924    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c5cdb566b-p7xlh"]
Sep 17 20:52:38.488960 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:38.488929    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c5cdb566b-p7xlh"]
Sep 17 20:52:38.669640 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:52:38.669607    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcab22c0-5a7b-45db-b464-657a40d83e0e" path="/var/lib/kubelet/pods/bcab22c0-5a7b-45db-b464-657a40d83e0e/volumes"
Sep 17 20:53:25.244219 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.244186    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b776ccfdf-xgd9c"]
Sep 17 20:53:25.244705 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.244541    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e7c0e30-282e-41d9-90bc-39f26a48444a" containerName="console"
Sep 17 20:53:25.244705 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.244570    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e7c0e30-282e-41d9-90bc-39f26a48444a" containerName="console"
Sep 17 20:53:25.244705 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.244579    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcab22c0-5a7b-45db-b464-657a40d83e0e" containerName="console"
Sep 17 20:53:25.244705 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.244584    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcab22c0-5a7b-45db-b464-657a40d83e0e" containerName="console"
Sep 17 20:53:25.244705 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.244600    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a49ced7b-edb8-4aa6-a993-ee060eba26d8" containerName="console"
Sep 17 20:53:25.244705 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.244606    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a49ced7b-edb8-4aa6-a993-ee060eba26d8" containerName="console"
Sep 17 20:53:25.244705 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.244656    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a49ced7b-edb8-4aa6-a993-ee060eba26d8" containerName="console"
Sep 17 20:53:25.244705 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.244663    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e7c0e30-282e-41d9-90bc-39f26a48444a" containerName="console"
Sep 17 20:53:25.244705 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.244669    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcab22c0-5a7b-45db-b464-657a40d83e0e" containerName="console"
Sep 17 20:53:25.247610 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.247594    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.262288 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.262263    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b776ccfdf-xgd9c"]
Sep 17 20:53:25.423687 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.423640    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-config\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.423687 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.423688    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-serving-cert\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.423934 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.423747    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-trusted-ca-bundle\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.423934 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.423781    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-oauth-config\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.423934 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.423803    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gnpp\" (UniqueName: \"kubernetes.io/projected/5e377f08-25ce-4700-b5bf-494d1acbf0a6-kube-api-access-9gnpp\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.423934 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.423837    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-service-ca\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.423934 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.423886    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-oauth-serving-cert\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.524672 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.524575    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-trusted-ca-bundle\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.524672 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.524620    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-oauth-config\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.524672 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.524638    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gnpp\" (UniqueName: \"kubernetes.io/projected/5e377f08-25ce-4700-b5bf-494d1acbf0a6-kube-api-access-9gnpp\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.524987 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.524798    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-service-ca\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.524987 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.524831    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-oauth-serving-cert\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.524987 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.524976    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-config\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.525150 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.525007    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-serving-cert\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.525578 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.525533    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-oauth-serving-cert\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.525714 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.525595    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-service-ca\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.525714 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.525648    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-trusted-ca-bundle\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.525714 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.525675    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-config\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.527334 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.527308    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-oauth-config\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.527495 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.527479    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-serving-cert\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.535256 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.535235    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gnpp\" (UniqueName: \"kubernetes.io/projected/5e377f08-25ce-4700-b5bf-494d1acbf0a6-kube-api-access-9gnpp\") pod \"console-5b776ccfdf-xgd9c\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") " pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.557280 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.557251    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:25.833169 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:25.833135    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b776ccfdf-xgd9c"]
Sep 17 20:53:25.836772 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:53:25.836742    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e377f08_25ce_4700_b5bf_494d1acbf0a6.slice/crio-d27469a81dcb1ac69e1af9c738e71d4d9fa861fb5cb4c46b87b4f4e0b4a58603 WatchSource:0}: Error finding container d27469a81dcb1ac69e1af9c738e71d4d9fa861fb5cb4c46b87b4f4e0b4a58603: Status 404 returned error can't find the container with id d27469a81dcb1ac69e1af9c738e71d4d9fa861fb5cb4c46b87b4f4e0b4a58603
Sep 17 20:53:26.653545 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:26.653510    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b776ccfdf-xgd9c" event={"ID":"5e377f08-25ce-4700-b5bf-494d1acbf0a6","Type":"ContainerStarted","Data":"9de00bb15e1a9997068f8d6f9a7a1f918c7e572fe8e3dae0a7765ae9809be4be"}
Sep 17 20:53:26.653939 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:26.653587    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b776ccfdf-xgd9c" event={"ID":"5e377f08-25ce-4700-b5bf-494d1acbf0a6","Type":"ContainerStarted","Data":"d27469a81dcb1ac69e1af9c738e71d4d9fa861fb5cb4c46b87b4f4e0b4a58603"}
Sep 17 20:53:26.672437 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:26.672389    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b776ccfdf-xgd9c" podStartSLOduration=1.672374166 podStartE2EDuration="1.672374166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:53:26.670985081 +0000 UTC m=+218.175462677" watchObservedRunningTime="2026-09-17 20:53:26.672374166 +0000 UTC m=+218.176851762"
Sep 17 20:53:35.558249 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:35.558213    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:35.558736 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:35.558312    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:35.563256 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:35.563236    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:35.700230 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:35.700200    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:53:35.753658 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:53:35.753628    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-675969869c-ltvsw"]
Sep 17 20:54:00.774075 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:00.773975    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-675969869c-ltvsw" podUID="5153c3b0-7122-4afc-8908-a44370ded276" containerName="console" containerID="cri-o://a52aa575e2a7bb771fbf12a9a82f9001b110a923b88803762b3ef3d8f6260bb8" gracePeriod=15
Sep 17 20:54:01.011061 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.011034    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:54:01.136031 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.136000    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-console-config\") pod \"5153c3b0-7122-4afc-8908-a44370ded276\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") "
Sep 17 20:54:01.136214 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.136060    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5153c3b0-7122-4afc-8908-a44370ded276-console-serving-cert\") pod \"5153c3b0-7122-4afc-8908-a44370ded276\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") "
Sep 17 20:54:01.136214 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.136078    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5153c3b0-7122-4afc-8908-a44370ded276-console-oauth-config\") pod \"5153c3b0-7122-4afc-8908-a44370ded276\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") "
Sep 17 20:54:01.136214 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.136098    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjfdv\" (UniqueName: \"kubernetes.io/projected/5153c3b0-7122-4afc-8908-a44370ded276-kube-api-access-bjfdv\") pod \"5153c3b0-7122-4afc-8908-a44370ded276\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") "
Sep 17 20:54:01.136214 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.136130    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-oauth-serving-cert\") pod \"5153c3b0-7122-4afc-8908-a44370ded276\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") "
Sep 17 20:54:01.136214 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.136162    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-trusted-ca-bundle\") pod \"5153c3b0-7122-4afc-8908-a44370ded276\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") "
Sep 17 20:54:01.136214 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.136186    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-service-ca\") pod \"5153c3b0-7122-4afc-8908-a44370ded276\" (UID: \"5153c3b0-7122-4afc-8908-a44370ded276\") "
Sep 17 20:54:01.136688 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.136519    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-console-config" (OuterVolumeSpecName: "console-config") pod "5153c3b0-7122-4afc-8908-a44370ded276" (UID: "5153c3b0-7122-4afc-8908-a44370ded276"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:54:01.136811 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.136691    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5153c3b0-7122-4afc-8908-a44370ded276" (UID: "5153c3b0-7122-4afc-8908-a44370ded276"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:54:01.136811 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.136718    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5153c3b0-7122-4afc-8908-a44370ded276" (UID: "5153c3b0-7122-4afc-8908-a44370ded276"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:54:01.136897 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.136819    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-service-ca" (OuterVolumeSpecName: "service-ca") pod "5153c3b0-7122-4afc-8908-a44370ded276" (UID: "5153c3b0-7122-4afc-8908-a44370ded276"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:54:01.138445 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.138420    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5153c3b0-7122-4afc-8908-a44370ded276-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5153c3b0-7122-4afc-8908-a44370ded276" (UID: "5153c3b0-7122-4afc-8908-a44370ded276"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:54:01.138626 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.138445    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5153c3b0-7122-4afc-8908-a44370ded276-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5153c3b0-7122-4afc-8908-a44370ded276" (UID: "5153c3b0-7122-4afc-8908-a44370ded276"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:54:01.138626 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.138454    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5153c3b0-7122-4afc-8908-a44370ded276-kube-api-access-bjfdv" (OuterVolumeSpecName: "kube-api-access-bjfdv") pod "5153c3b0-7122-4afc-8908-a44370ded276" (UID: "5153c3b0-7122-4afc-8908-a44370ded276"). InnerVolumeSpecName "kube-api-access-bjfdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:54:01.237477 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.237425    2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-console-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:54:01.237477 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.237467    2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5153c3b0-7122-4afc-8908-a44370ded276-console-serving-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:54:01.237477 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.237477    2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5153c3b0-7122-4afc-8908-a44370ded276-console-oauth-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:54:01.237477 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.237486    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjfdv\" (UniqueName: \"kubernetes.io/projected/5153c3b0-7122-4afc-8908-a44370ded276-kube-api-access-bjfdv\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:54:01.237477 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.237495    2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-oauth-serving-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:54:01.237800 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.237503    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-trusted-ca-bundle\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:54:01.237800 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.237511    2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5153c3b0-7122-4afc-8908-a44370ded276-service-ca\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:54:01.801123 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.801089    2571 generic.go:356] "Generic (PLEG): container finished" podID="5153c3b0-7122-4afc-8908-a44370ded276" containerID="a52aa575e2a7bb771fbf12a9a82f9001b110a923b88803762b3ef3d8f6260bb8" exitCode=0
Sep 17 20:54:01.801650 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.801134    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-675969869c-ltvsw"
Sep 17 20:54:01.801650 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.801157    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-675969869c-ltvsw" event={"ID":"5153c3b0-7122-4afc-8908-a44370ded276","Type":"ContainerDied","Data":"a52aa575e2a7bb771fbf12a9a82f9001b110a923b88803762b3ef3d8f6260bb8"}
Sep 17 20:54:01.801650 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.801206    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-675969869c-ltvsw" event={"ID":"5153c3b0-7122-4afc-8908-a44370ded276","Type":"ContainerDied","Data":"b6f2b87c500619be81d7df0c06f981fb707ebce3fbf01ecab2c888251f55abcc"}
Sep 17 20:54:01.801650 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.801236    2571 scope.go:117] "RemoveContainer" containerID="a52aa575e2a7bb771fbf12a9a82f9001b110a923b88803762b3ef3d8f6260bb8"
Sep 17 20:54:01.810091 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.810076    2571 scope.go:117] "RemoveContainer" containerID="a52aa575e2a7bb771fbf12a9a82f9001b110a923b88803762b3ef3d8f6260bb8"
Sep 17 20:54:01.810389 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:54:01.810369    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52aa575e2a7bb771fbf12a9a82f9001b110a923b88803762b3ef3d8f6260bb8\": container with ID starting with a52aa575e2a7bb771fbf12a9a82f9001b110a923b88803762b3ef3d8f6260bb8 not found: ID does not exist" containerID="a52aa575e2a7bb771fbf12a9a82f9001b110a923b88803762b3ef3d8f6260bb8"
Sep 17 20:54:01.810459 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.810400    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52aa575e2a7bb771fbf12a9a82f9001b110a923b88803762b3ef3d8f6260bb8"} err="failed to get container status \"a52aa575e2a7bb771fbf12a9a82f9001b110a923b88803762b3ef3d8f6260bb8\": rpc error: code = NotFound desc = could not find container \"a52aa575e2a7bb771fbf12a9a82f9001b110a923b88803762b3ef3d8f6260bb8\": container with ID starting with a52aa575e2a7bb771fbf12a9a82f9001b110a923b88803762b3ef3d8f6260bb8 not found: ID does not exist"
Sep 17 20:54:01.822486 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.822459    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-675969869c-ltvsw"]
Sep 17 20:54:01.828512 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:01.828489    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-675969869c-ltvsw"]
Sep 17 20:54:02.668741 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:02.668701    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5153c3b0-7122-4afc-8908-a44370ded276" path="/var/lib/kubelet/pods/5153c3b0-7122-4afc-8908-a44370ded276/volumes"
Sep 17 20:54:50.563204 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:54:50.563177    2571 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 20:55:03.830156 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:03.830120    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb"]
Sep 17 20:55:03.832471 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:03.830471    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5153c3b0-7122-4afc-8908-a44370ded276" containerName="console"
Sep 17 20:55:03.832471 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:03.830488    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5153c3b0-7122-4afc-8908-a44370ded276" containerName="console"
Sep 17 20:55:03.832471 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:03.830566    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5153c3b0-7122-4afc-8908-a44370ded276" containerName="console"
Sep 17 20:55:03.833419 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:03.833405    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb"
Sep 17 20:55:03.836159 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:03.836128    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 20:55:03.836266 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:03.836175    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kqqgr\""
Sep 17 20:55:03.837106 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:03.837089    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 20:55:03.843114 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:03.843092    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb"]
Sep 17 20:55:03.956795 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:03.956752    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54ebad00-969b-4b70-a9bd-125a208a51d1-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb\" (UID: \"54ebad00-969b-4b70-a9bd-125a208a51d1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb"
Sep 17 20:55:03.956991 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:03.956838    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7w85\" (UniqueName: \"kubernetes.io/projected/54ebad00-969b-4b70-a9bd-125a208a51d1-kube-api-access-c7w85\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb\" (UID: \"54ebad00-969b-4b70-a9bd-125a208a51d1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb"
Sep 17 20:55:03.956991 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:03.956877    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54ebad00-969b-4b70-a9bd-125a208a51d1-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb\" (UID: \"54ebad00-969b-4b70-a9bd-125a208a51d1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb"
Sep 17 20:55:04.057639 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:04.057605    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54ebad00-969b-4b70-a9bd-125a208a51d1-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb\" (UID: \"54ebad00-969b-4b70-a9bd-125a208a51d1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb"
Sep 17 20:55:04.057818 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:04.057669    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7w85\" (UniqueName: \"kubernetes.io/projected/54ebad00-969b-4b70-a9bd-125a208a51d1-kube-api-access-c7w85\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb\" (UID: \"54ebad00-969b-4b70-a9bd-125a208a51d1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb"
Sep 17 20:55:04.057818 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:04.057697    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54ebad00-969b-4b70-a9bd-125a208a51d1-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb\" (UID: \"54ebad00-969b-4b70-a9bd-125a208a51d1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb"
Sep 17 20:55:04.058005 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:04.057986    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54ebad00-969b-4b70-a9bd-125a208a51d1-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb\" (UID: \"54ebad00-969b-4b70-a9bd-125a208a51d1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb"
Sep 17 20:55:04.058047 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:04.058013    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54ebad00-969b-4b70-a9bd-125a208a51d1-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb\" (UID: \"54ebad00-969b-4b70-a9bd-125a208a51d1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb"
Sep 17 20:55:04.067697 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:04.067668    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7w85\" (UniqueName: \"kubernetes.io/projected/54ebad00-969b-4b70-a9bd-125a208a51d1-kube-api-access-c7w85\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb\" (UID: \"54ebad00-969b-4b70-a9bd-125a208a51d1\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb"
Sep 17 20:55:04.142716 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:04.142627    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb"
Sep 17 20:55:04.421043 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:04.420968    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb"]
Sep 17 20:55:04.425111 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:55:04.425080    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ebad00_969b_4b70_a9bd_125a208a51d1.slice/crio-e34f4eb6cc8267b10d7cc0115916bafa1d300a87f7ed10af78dc459ee4e36297 WatchSource:0}: Error finding container e34f4eb6cc8267b10d7cc0115916bafa1d300a87f7ed10af78dc459ee4e36297: Status 404 returned error can't find the container with id e34f4eb6cc8267b10d7cc0115916bafa1d300a87f7ed10af78dc459ee4e36297
Sep 17 20:55:04.427151 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:04.427138    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 20:55:05.056010 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:05.055978    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb" event={"ID":"54ebad00-969b-4b70-a9bd-125a208a51d1","Type":"ContainerStarted","Data":"e34f4eb6cc8267b10d7cc0115916bafa1d300a87f7ed10af78dc459ee4e36297"}
Sep 17 20:55:11.082734 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:11.082704    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb" event={"ID":"54ebad00-969b-4b70-a9bd-125a208a51d1","Type":"ContainerStarted","Data":"c6774a05479fadbf5f1a6b1727063fceb1b836f561f3642ce21d9f8b3b79b242"}
Sep 17 20:55:12.087718 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:12.087682    2571 generic.go:356] "Generic (PLEG): container finished" podID="54ebad00-969b-4b70-a9bd-125a208a51d1" containerID="c6774a05479fadbf5f1a6b1727063fceb1b836f561f3642ce21d9f8b3b79b242" exitCode=0
Sep 17 20:55:12.088201 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:12.087740    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb" event={"ID":"54ebad00-969b-4b70-a9bd-125a208a51d1","Type":"ContainerDied","Data":"c6774a05479fadbf5f1a6b1727063fceb1b836f561f3642ce21d9f8b3b79b242"}
Sep 17 20:55:14.097980 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:14.097952    2571 generic.go:356] "Generic (PLEG): container finished" podID="54ebad00-969b-4b70-a9bd-125a208a51d1" containerID="a83a731b16076f46d4f5a7642acb46d2cd03d38e5e500aff6a12c3cb40bdbf36" exitCode=0
Sep 17 20:55:14.098337 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:14.098027    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb" event={"ID":"54ebad00-969b-4b70-a9bd-125a208a51d1","Type":"ContainerDied","Data":"a83a731b16076f46d4f5a7642acb46d2cd03d38e5e500aff6a12c3cb40bdbf36"}
Sep 17 20:55:21.139073 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:21.139037    2571 generic.go:356] "Generic (PLEG): container finished" podID="54ebad00-969b-4b70-a9bd-125a208a51d1" containerID="e5dace540fa45fa1ed1e24e51e6b116763d44fea8981d213d8b73a271e7bd984" exitCode=0
Sep 17 20:55:21.139485 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:21.139108    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb" event={"ID":"54ebad00-969b-4b70-a9bd-125a208a51d1","Type":"ContainerDied","Data":"e5dace540fa45fa1ed1e24e51e6b116763d44fea8981d213d8b73a271e7bd984"}
Sep 17 20:55:22.271523 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:22.271493    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb"
Sep 17 20:55:22.308355 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:22.308323    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54ebad00-969b-4b70-a9bd-125a208a51d1-util\") pod \"54ebad00-969b-4b70-a9bd-125a208a51d1\" (UID: \"54ebad00-969b-4b70-a9bd-125a208a51d1\") "
Sep 17 20:55:22.308528 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:22.308412    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7w85\" (UniqueName: \"kubernetes.io/projected/54ebad00-969b-4b70-a9bd-125a208a51d1-kube-api-access-c7w85\") pod \"54ebad00-969b-4b70-a9bd-125a208a51d1\" (UID: \"54ebad00-969b-4b70-a9bd-125a208a51d1\") "
Sep 17 20:55:22.308528 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:22.308524    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54ebad00-969b-4b70-a9bd-125a208a51d1-bundle\") pod \"54ebad00-969b-4b70-a9bd-125a208a51d1\" (UID: \"54ebad00-969b-4b70-a9bd-125a208a51d1\") "
Sep 17 20:55:22.309384 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:22.309344    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ebad00-969b-4b70-a9bd-125a208a51d1-bundle" (OuterVolumeSpecName: "bundle") pod "54ebad00-969b-4b70-a9bd-125a208a51d1" (UID: "54ebad00-969b-4b70-a9bd-125a208a51d1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:55:22.310898 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:22.310871    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ebad00-969b-4b70-a9bd-125a208a51d1-kube-api-access-c7w85" (OuterVolumeSpecName: "kube-api-access-c7w85") pod "54ebad00-969b-4b70-a9bd-125a208a51d1" (UID: "54ebad00-969b-4b70-a9bd-125a208a51d1"). InnerVolumeSpecName "kube-api-access-c7w85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:55:22.313313 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:22.313291    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ebad00-969b-4b70-a9bd-125a208a51d1-util" (OuterVolumeSpecName: "util") pod "54ebad00-969b-4b70-a9bd-125a208a51d1" (UID: "54ebad00-969b-4b70-a9bd-125a208a51d1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:55:22.409955 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:22.409854    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7w85\" (UniqueName: \"kubernetes.io/projected/54ebad00-969b-4b70-a9bd-125a208a51d1-kube-api-access-c7w85\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:55:22.409955 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:22.409898    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54ebad00-969b-4b70-a9bd-125a208a51d1-bundle\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:55:22.409955 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:22.409909    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54ebad00-969b-4b70-a9bd-125a208a51d1-util\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:55:23.149018 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:23.148996    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb"
Sep 17 20:55:23.149190 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:23.148994    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adwjrb" event={"ID":"54ebad00-969b-4b70-a9bd-125a208a51d1","Type":"ContainerDied","Data":"e34f4eb6cc8267b10d7cc0115916bafa1d300a87f7ed10af78dc459ee4e36297"}
Sep 17 20:55:23.149190 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:23.149110    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e34f4eb6cc8267b10d7cc0115916bafa1d300a87f7ed10af78dc459ee4e36297"
Sep 17 20:55:27.545496 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.545463    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bdx5x"]
Sep 17 20:55:27.545889 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.545817    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ebad00-969b-4b70-a9bd-125a208a51d1" containerName="util"
Sep 17 20:55:27.545889 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.545828    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ebad00-969b-4b70-a9bd-125a208a51d1" containerName="util"
Sep 17 20:55:27.545889 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.545844    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ebad00-969b-4b70-a9bd-125a208a51d1" containerName="extract"
Sep 17 20:55:27.545889 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.545849    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ebad00-969b-4b70-a9bd-125a208a51d1" containerName="extract"
Sep 17 20:55:27.545889 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.545856    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ebad00-969b-4b70-a9bd-125a208a51d1" containerName="pull"
Sep 17 20:55:27.545889 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.545862    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ebad00-969b-4b70-a9bd-125a208a51d1" containerName="pull"
Sep 17 20:55:27.546068 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.545922    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ebad00-969b-4b70-a9bd-125a208a51d1" containerName="extract"
Sep 17 20:55:27.549006 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.548991    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bdx5x"
Sep 17 20:55:27.551684 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.551657    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 20:55:27.551684 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.551664    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-8q4l7\""
Sep 17 20:55:27.552125 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.552112    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 20:55:27.552209 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.552195    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 20:55:27.567287 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.567263    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bdx5x"]
Sep 17 20:55:27.652037 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.651999    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/7f405375-b762-4ef9-8a3b-a2b6f440ef31-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-bdx5x\" (UID: \"7f405375-b762-4ef9-8a3b-a2b6f440ef31\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bdx5x"
Sep 17 20:55:27.652213 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.652079    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58srx\" (UniqueName: \"kubernetes.io/projected/7f405375-b762-4ef9-8a3b-a2b6f440ef31-kube-api-access-58srx\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-bdx5x\" (UID: \"7f405375-b762-4ef9-8a3b-a2b6f440ef31\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bdx5x"
Sep 17 20:55:27.752507 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.752471    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/7f405375-b762-4ef9-8a3b-a2b6f440ef31-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-bdx5x\" (UID: \"7f405375-b762-4ef9-8a3b-a2b6f440ef31\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bdx5x"
Sep 17 20:55:27.752711 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.752544    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58srx\" (UniqueName: \"kubernetes.io/projected/7f405375-b762-4ef9-8a3b-a2b6f440ef31-kube-api-access-58srx\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-bdx5x\" (UID: \"7f405375-b762-4ef9-8a3b-a2b6f440ef31\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bdx5x"
Sep 17 20:55:27.755059 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.755040    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/7f405375-b762-4ef9-8a3b-a2b6f440ef31-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-bdx5x\" (UID: \"7f405375-b762-4ef9-8a3b-a2b6f440ef31\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bdx5x"
Sep 17 20:55:27.765940 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.765913    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58srx\" (UniqueName: \"kubernetes.io/projected/7f405375-b762-4ef9-8a3b-a2b6f440ef31-kube-api-access-58srx\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-bdx5x\" (UID: \"7f405375-b762-4ef9-8a3b-a2b6f440ef31\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bdx5x"
Sep 17 20:55:27.858202 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:27.858172    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bdx5x"
Sep 17 20:55:28.153883 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:28.153808    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bdx5x"]
Sep 17 20:55:28.158184 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:55:28.158156    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f405375_b762_4ef9_8a3b_a2b6f440ef31.slice/crio-623ee40c36aa3ffed8d924d1badda3f3dba80ab2a196d2a1830a56b1325e051e WatchSource:0}: Error finding container 623ee40c36aa3ffed8d924d1badda3f3dba80ab2a196d2a1830a56b1325e051e: Status 404 returned error can't find the container with id 623ee40c36aa3ffed8d924d1badda3f3dba80ab2a196d2a1830a56b1325e051e
Sep 17 20:55:28.172896 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:28.172865    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bdx5x" event={"ID":"7f405375-b762-4ef9-8a3b-a2b6f440ef31","Type":"ContainerStarted","Data":"623ee40c36aa3ffed8d924d1badda3f3dba80ab2a196d2a1830a56b1325e051e"}
Sep 17 20:55:31.195526 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.195441    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bdx5x" event={"ID":"7f405375-b762-4ef9-8a3b-a2b6f440ef31","Type":"ContainerStarted","Data":"d652b9a92703981898e34d7f462a5785c614a37150f749825e40fa0ba5455896"}
Sep 17 20:55:31.195914 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.195620    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bdx5x"
Sep 17 20:55:31.223062 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.223015    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bdx5x" podStartSLOduration=1.47348459 podStartE2EDuration="4.222999879s" totalImagesPullingTime="2.749515289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:55:31.222675868 +0000 UTC m=+342.727153463" watchObservedRunningTime="2026-09-17 20:55:31.222999879 +0000 UTC m=+342.727477473"
Sep 17 20:55:31.711974 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.711938    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-j5mbx"]
Sep 17 20:55:31.715110 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.715095    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-j5mbx"
Sep 17 20:55:31.718296 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.718262    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 20:55:31.718441 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.718308    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-gw6s5\""
Sep 17 20:55:31.718441 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.718262    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 20:55:31.730437 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.730414    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-j5mbx"]
Sep 17 20:55:31.789525 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.789488    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/40f44a0d-3915-4614-a2c7-d01d6dc851db-cabundle0\") pod \"keda-operator-579c8958f7-j5mbx\" (UID: \"40f44a0d-3915-4614-a2c7-d01d6dc851db\") " pod="openshift-keda/keda-operator-579c8958f7-j5mbx"
Sep 17 20:55:31.789730 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.789542    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/40f44a0d-3915-4614-a2c7-d01d6dc851db-certificates\") pod \"keda-operator-579c8958f7-j5mbx\" (UID: \"40f44a0d-3915-4614-a2c7-d01d6dc851db\") " pod="openshift-keda/keda-operator-579c8958f7-j5mbx"
Sep 17 20:55:31.789730 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.789642    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmttq\" (UniqueName: \"kubernetes.io/projected/40f44a0d-3915-4614-a2c7-d01d6dc851db-kube-api-access-zmttq\") pod \"keda-operator-579c8958f7-j5mbx\" (UID: \"40f44a0d-3915-4614-a2c7-d01d6dc851db\") " pod="openshift-keda/keda-operator-579c8958f7-j5mbx"
Sep 17 20:55:31.890401 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.890369    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmttq\" (UniqueName: \"kubernetes.io/projected/40f44a0d-3915-4614-a2c7-d01d6dc851db-kube-api-access-zmttq\") pod \"keda-operator-579c8958f7-j5mbx\" (UID: \"40f44a0d-3915-4614-a2c7-d01d6dc851db\") " pod="openshift-keda/keda-operator-579c8958f7-j5mbx"
Sep 17 20:55:31.890585 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.890457    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/40f44a0d-3915-4614-a2c7-d01d6dc851db-cabundle0\") pod \"keda-operator-579c8958f7-j5mbx\" (UID: \"40f44a0d-3915-4614-a2c7-d01d6dc851db\") " pod="openshift-keda/keda-operator-579c8958f7-j5mbx"
Sep 17 20:55:31.890585 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.890503    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/40f44a0d-3915-4614-a2c7-d01d6dc851db-certificates\") pod \"keda-operator-579c8958f7-j5mbx\" (UID: \"40f44a0d-3915-4614-a2c7-d01d6dc851db\") " pod="openshift-keda/keda-operator-579c8958f7-j5mbx"
Sep 17 20:55:31.890669 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:55:31.890617    2571 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 17 20:55:31.890669 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:55:31.890636    2571 secret.go:281] references non-existent secret key: ca.crt
Sep 17 20:55:31.890669 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:55:31.890647    2571 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 20:55:31.890669 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:55:31.890661    2571 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-j5mbx: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 20:55:31.890784 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:55:31.890721    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40f44a0d-3915-4614-a2c7-d01d6dc851db-certificates podName:40f44a0d-3915-4614-a2c7-d01d6dc851db nodeName:}" failed. No retries permitted until 2026-09-17 20:55:32.390702041 +0000 UTC m=+343.895179629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/40f44a0d-3915-4614-a2c7-d01d6dc851db-certificates") pod "keda-operator-579c8958f7-j5mbx" (UID: "40f44a0d-3915-4614-a2c7-d01d6dc851db") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 20:55:31.891144 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.891124    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/40f44a0d-3915-4614-a2c7-d01d6dc851db-cabundle0\") pod \"keda-operator-579c8958f7-j5mbx\" (UID: \"40f44a0d-3915-4614-a2c7-d01d6dc851db\") " pod="openshift-keda/keda-operator-579c8958f7-j5mbx"
Sep 17 20:55:31.900193 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:31.900168    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmttq\" (UniqueName: \"kubernetes.io/projected/40f44a0d-3915-4614-a2c7-d01d6dc851db-kube-api-access-zmttq\") pod \"keda-operator-579c8958f7-j5mbx\" (UID: \"40f44a0d-3915-4614-a2c7-d01d6dc851db\") " pod="openshift-keda/keda-operator-579c8958f7-j5mbx"
Sep 17 20:55:32.350265 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.350229    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"]
Sep 17 20:55:32.353508 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.353487    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"
Sep 17 20:55:32.355846 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.355826    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 20:55:32.364041 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.364010    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"]
Sep 17 20:55:32.394665 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.394623    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dk7j\" (UniqueName: \"kubernetes.io/projected/4b27ad67-b01d-484e-8389-4a8e3f27a2d1-kube-api-access-5dk7j\") pod \"keda-metrics-apiserver-77857b7b98-69t7b\" (UID: \"4b27ad67-b01d-484e-8389-4a8e3f27a2d1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"
Sep 17 20:55:32.394830 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.394694    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4b27ad67-b01d-484e-8389-4a8e3f27a2d1-certificates\") pod \"keda-metrics-apiserver-77857b7b98-69t7b\" (UID: \"4b27ad67-b01d-484e-8389-4a8e3f27a2d1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"
Sep 17 20:55:32.394830 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.394747    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/40f44a0d-3915-4614-a2c7-d01d6dc851db-certificates\") pod \"keda-operator-579c8958f7-j5mbx\" (UID: \"40f44a0d-3915-4614-a2c7-d01d6dc851db\") " pod="openshift-keda/keda-operator-579c8958f7-j5mbx"
Sep 17 20:55:32.394909 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.394830    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4b27ad67-b01d-484e-8389-4a8e3f27a2d1-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-69t7b\" (UID: \"4b27ad67-b01d-484e-8389-4a8e3f27a2d1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"
Sep 17 20:55:32.397304 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.397282    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/40f44a0d-3915-4614-a2c7-d01d6dc851db-certificates\") pod \"keda-operator-579c8958f7-j5mbx\" (UID: \"40f44a0d-3915-4614-a2c7-d01d6dc851db\") " pod="openshift-keda/keda-operator-579c8958f7-j5mbx"
Sep 17 20:55:32.495599 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.495525    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4b27ad67-b01d-484e-8389-4a8e3f27a2d1-certificates\") pod \"keda-metrics-apiserver-77857b7b98-69t7b\" (UID: \"4b27ad67-b01d-484e-8389-4a8e3f27a2d1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"
Sep 17 20:55:32.495780 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.495643    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4b27ad67-b01d-484e-8389-4a8e3f27a2d1-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-69t7b\" (UID: \"4b27ad67-b01d-484e-8389-4a8e3f27a2d1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"
Sep 17 20:55:32.495780 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.495674    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dk7j\" (UniqueName: \"kubernetes.io/projected/4b27ad67-b01d-484e-8389-4a8e3f27a2d1-kube-api-access-5dk7j\") pod \"keda-metrics-apiserver-77857b7b98-69t7b\" (UID: \"4b27ad67-b01d-484e-8389-4a8e3f27a2d1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"
Sep 17 20:55:32.495780 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:55:32.495682    2571 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 17 20:55:32.495780 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:55:32.495710    2571 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b: secret "keda-metrics-apiserver-certs" not found
Sep 17 20:55:32.495780 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:55:32.495768    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b27ad67-b01d-484e-8389-4a8e3f27a2d1-certificates podName:4b27ad67-b01d-484e-8389-4a8e3f27a2d1 nodeName:}" failed. No retries permitted until 2026-09-17 20:55:32.995751695 +0000 UTC m=+344.500229269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4b27ad67-b01d-484e-8389-4a8e3f27a2d1-certificates") pod "keda-metrics-apiserver-77857b7b98-69t7b" (UID: "4b27ad67-b01d-484e-8389-4a8e3f27a2d1") : secret "keda-metrics-apiserver-certs" not found
Sep 17 20:55:32.496055 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.496021    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4b27ad67-b01d-484e-8389-4a8e3f27a2d1-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-69t7b\" (UID: \"4b27ad67-b01d-484e-8389-4a8e3f27a2d1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"
Sep 17 20:55:32.507038 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.507011    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dk7j\" (UniqueName: \"kubernetes.io/projected/4b27ad67-b01d-484e-8389-4a8e3f27a2d1-kube-api-access-5dk7j\") pod \"keda-metrics-apiserver-77857b7b98-69t7b\" (UID: \"4b27ad67-b01d-484e-8389-4a8e3f27a2d1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"
Sep 17 20:55:32.553581 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.553536    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-kbndh"]
Sep 17 20:55:32.556835 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.556816    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-kbndh"
Sep 17 20:55:32.559572 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.559539    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 20:55:32.567206 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.567183    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-kbndh"]
Sep 17 20:55:32.596361 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.596327    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aa784099-3ffe-4dc5-8d96-28f07fe85744-certificates\") pod \"keda-admission-6c4d864b86-kbndh\" (UID: \"aa784099-3ffe-4dc5-8d96-28f07fe85744\") " pod="openshift-keda/keda-admission-6c4d864b86-kbndh"
Sep 17 20:55:32.596539 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.596415    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr82x\" (UniqueName: \"kubernetes.io/projected/aa784099-3ffe-4dc5-8d96-28f07fe85744-kube-api-access-cr82x\") pod \"keda-admission-6c4d864b86-kbndh\" (UID: \"aa784099-3ffe-4dc5-8d96-28f07fe85744\") " pod="openshift-keda/keda-admission-6c4d864b86-kbndh"
Sep 17 20:55:32.625255 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.625185    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-j5mbx"
Sep 17 20:55:32.697105 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.697072    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aa784099-3ffe-4dc5-8d96-28f07fe85744-certificates\") pod \"keda-admission-6c4d864b86-kbndh\" (UID: \"aa784099-3ffe-4dc5-8d96-28f07fe85744\") " pod="openshift-keda/keda-admission-6c4d864b86-kbndh"
Sep 17 20:55:32.697227 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.697130    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr82x\" (UniqueName: \"kubernetes.io/projected/aa784099-3ffe-4dc5-8d96-28f07fe85744-kube-api-access-cr82x\") pod \"keda-admission-6c4d864b86-kbndh\" (UID: \"aa784099-3ffe-4dc5-8d96-28f07fe85744\") " pod="openshift-keda/keda-admission-6c4d864b86-kbndh"
Sep 17 20:55:32.697439 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:55:32.697398    2571 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 17 20:55:32.697439 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:55:32.697433    2571 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-kbndh: secret "keda-admission-webhooks-certs" not found
Sep 17 20:55:32.697620 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:55:32.697501    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa784099-3ffe-4dc5-8d96-28f07fe85744-certificates podName:aa784099-3ffe-4dc5-8d96-28f07fe85744 nodeName:}" failed. No retries permitted until 2026-09-17 20:55:33.197480479 +0000 UTC m=+344.701958058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/aa784099-3ffe-4dc5-8d96-28f07fe85744-certificates") pod "keda-admission-6c4d864b86-kbndh" (UID: "aa784099-3ffe-4dc5-8d96-28f07fe85744") : secret "keda-admission-webhooks-certs" not found
Sep 17 20:55:32.708309 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.708279    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr82x\" (UniqueName: \"kubernetes.io/projected/aa784099-3ffe-4dc5-8d96-28f07fe85744-kube-api-access-cr82x\") pod \"keda-admission-6c4d864b86-kbndh\" (UID: \"aa784099-3ffe-4dc5-8d96-28f07fe85744\") " pod="openshift-keda/keda-admission-6c4d864b86-kbndh"
Sep 17 20:55:32.905968 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:32.905943    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-j5mbx"]
Sep 17 20:55:32.908532 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:55:32.908511    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f44a0d_3915_4614_a2c7_d01d6dc851db.slice/crio-20fbacb4277d10e4826581c939d2590c04a5adaacf5b3b726dccabc76f17c170 WatchSource:0}: Error finding container 20fbacb4277d10e4826581c939d2590c04a5adaacf5b3b726dccabc76f17c170: Status 404 returned error can't find the container with id 20fbacb4277d10e4826581c939d2590c04a5adaacf5b3b726dccabc76f17c170
Sep 17 20:55:33.000034 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:33.000000    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4b27ad67-b01d-484e-8389-4a8e3f27a2d1-certificates\") pod \"keda-metrics-apiserver-77857b7b98-69t7b\" (UID: \"4b27ad67-b01d-484e-8389-4a8e3f27a2d1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"
Sep 17 20:55:33.002740 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:33.002718    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4b27ad67-b01d-484e-8389-4a8e3f27a2d1-certificates\") pod \"keda-metrics-apiserver-77857b7b98-69t7b\" (UID: \"4b27ad67-b01d-484e-8389-4a8e3f27a2d1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"
Sep 17 20:55:33.202680 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:33.202586    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aa784099-3ffe-4dc5-8d96-28f07fe85744-certificates\") pod \"keda-admission-6c4d864b86-kbndh\" (UID: \"aa784099-3ffe-4dc5-8d96-28f07fe85744\") " pod="openshift-keda/keda-admission-6c4d864b86-kbndh"
Sep 17 20:55:33.205340 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:33.205316    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/aa784099-3ffe-4dc5-8d96-28f07fe85744-certificates\") pod \"keda-admission-6c4d864b86-kbndh\" (UID: \"aa784099-3ffe-4dc5-8d96-28f07fe85744\") " pod="openshift-keda/keda-admission-6c4d864b86-kbndh"
Sep 17 20:55:33.208123 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:33.208099    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-j5mbx" event={"ID":"40f44a0d-3915-4614-a2c7-d01d6dc851db","Type":"ContainerStarted","Data":"20fbacb4277d10e4826581c939d2590c04a5adaacf5b3b726dccabc76f17c170"}
Sep 17 20:55:33.265477 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:33.265439    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"
Sep 17 20:55:33.468019 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:33.467929    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-kbndh"
Sep 17 20:55:33.549082 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:33.549053    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"]
Sep 17 20:55:33.551715 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:55:33.551684    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b27ad67_b01d_484e_8389_4a8e3f27a2d1.slice/crio-8b90f9103b25afa10689ac029760b782c1bf8415dbb3f57b2dbbba8bbd8a66ae WatchSource:0}: Error finding container 8b90f9103b25afa10689ac029760b782c1bf8415dbb3f57b2dbbba8bbd8a66ae: Status 404 returned error can't find the container with id 8b90f9103b25afa10689ac029760b782c1bf8415dbb3f57b2dbbba8bbd8a66ae
Sep 17 20:55:33.754296 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:33.754267    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-kbndh"]
Sep 17 20:55:33.757154 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:55:33.757121    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa784099_3ffe_4dc5_8d96_28f07fe85744.slice/crio-70b10cc77548b84164cdc1f3fda025d9ce6620c25d56f88da104ebc1b8d9b5bf WatchSource:0}: Error finding container 70b10cc77548b84164cdc1f3fda025d9ce6620c25d56f88da104ebc1b8d9b5bf: Status 404 returned error can't find the container with id 70b10cc77548b84164cdc1f3fda025d9ce6620c25d56f88da104ebc1b8d9b5bf
Sep 17 20:55:34.217411 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:34.217342    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-kbndh" event={"ID":"aa784099-3ffe-4dc5-8d96-28f07fe85744","Type":"ContainerStarted","Data":"70b10cc77548b84164cdc1f3fda025d9ce6620c25d56f88da104ebc1b8d9b5bf"}
Sep 17 20:55:34.219079 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:34.219024    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b" event={"ID":"4b27ad67-b01d-484e-8389-4a8e3f27a2d1","Type":"ContainerStarted","Data":"8b90f9103b25afa10689ac029760b782c1bf8415dbb3f57b2dbbba8bbd8a66ae"}
Sep 17 20:55:37.238033 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:37.237993    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b" event={"ID":"4b27ad67-b01d-484e-8389-4a8e3f27a2d1","Type":"ContainerStarted","Data":"96c893cbc1dae271754c52523a79b3a924791856838027f93accf523a6e5836b"}
Sep 17 20:55:37.238513 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:37.238101    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"
Sep 17 20:55:37.240418 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:37.240344    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-kbndh" event={"ID":"aa784099-3ffe-4dc5-8d96-28f07fe85744","Type":"ContainerStarted","Data":"ee55d90a9527ab0ab377de16fcef6a24970b8f956028cd828f18c367af8699a6"}
Sep 17 20:55:37.240418 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:37.240417    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-kbndh"
Sep 17 20:55:37.242655 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:37.242626    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-j5mbx" event={"ID":"40f44a0d-3915-4614-a2c7-d01d6dc851db","Type":"ContainerStarted","Data":"5321c197ae33d970d11c85cd66e501248fe00dfcd2ae900e91c9d8799154aa04"}
Sep 17 20:55:37.242788 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:37.242700    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-j5mbx"
Sep 17 20:55:37.257694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:37.257637    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b" podStartSLOduration=1.672668598 podStartE2EDuration="5.257619965s" totalImagesPullingTime="3.584951367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:55:37.255587869 +0000 UTC m=+348.760065474" watchObservedRunningTime="2026-09-17 20:55:37.257619965 +0000 UTC m=+348.762097563"
Sep 17 20:55:37.274596 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:37.274529    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-j5mbx" podStartSLOduration=2.055700849 podStartE2EDuration="6.274509716s" totalImagesPullingTime="4.218808867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:55:37.27242656 +0000 UTC m=+348.776904190" watchObservedRunningTime="2026-09-17 20:55:37.274509716 +0000 UTC m=+348.778987313"
Sep 17 20:55:37.290045 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:37.289997    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-kbndh" podStartSLOduration=1.920692015 podStartE2EDuration="5.289979524s" totalImagesPullingTime="3.369287509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:55:37.289538564 +0000 UTC m=+348.794016165" watchObservedRunningTime="2026-09-17 20:55:37.289979524 +0000 UTC m=+348.794457122"
Sep 17 20:55:48.252984 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:48.252957    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-69t7b"
Sep 17 20:55:52.203798 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:52.203766    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bdx5x"
Sep 17 20:55:58.250715 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:58.250684    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-j5mbx"
Sep 17 20:55:58.251088 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:55:58.250852    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-kbndh"
Sep 17 20:56:39.546637 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.546601    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-677fdbbb87-4h64j"]
Sep 17 20:56:39.548974 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.548955    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.572462 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.572435    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-677fdbbb87-4h64j"]
Sep 17 20:56:39.697504 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.697464    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24ff5f9f-91c5-4927-910c-974c795bd13b-oauth-serving-cert\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.697504 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.697510    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24ff5f9f-91c5-4927-910c-974c795bd13b-console-oauth-config\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.697754 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.697531    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24ff5f9f-91c5-4927-910c-974c795bd13b-service-ca\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.697754 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.697606    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ff5f9f-91c5-4927-910c-974c795bd13b-trusted-ca-bundle\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.697754 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.697633    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwpr6\" (UniqueName: \"kubernetes.io/projected/24ff5f9f-91c5-4927-910c-974c795bd13b-kube-api-access-jwpr6\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.697754 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.697662    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24ff5f9f-91c5-4927-910c-974c795bd13b-console-config\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.697754 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.697682    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24ff5f9f-91c5-4927-910c-974c795bd13b-console-serving-cert\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.799116 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.799005    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24ff5f9f-91c5-4927-910c-974c795bd13b-console-config\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.799116 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.799066    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24ff5f9f-91c5-4927-910c-974c795bd13b-console-serving-cert\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.799116 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.799106    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24ff5f9f-91c5-4927-910c-974c795bd13b-oauth-serving-cert\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.799466 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.799151    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24ff5f9f-91c5-4927-910c-974c795bd13b-console-oauth-config\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.799466 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.799180    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24ff5f9f-91c5-4927-910c-974c795bd13b-service-ca\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.799466 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.799222    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ff5f9f-91c5-4927-910c-974c795bd13b-trusted-ca-bundle\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.799466 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.799246    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwpr6\" (UniqueName: \"kubernetes.io/projected/24ff5f9f-91c5-4927-910c-974c795bd13b-kube-api-access-jwpr6\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.799947 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.799920    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24ff5f9f-91c5-4927-910c-974c795bd13b-console-config\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.800333 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.800302    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24ff5f9f-91c5-4927-910c-974c795bd13b-oauth-serving-cert\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.800492 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.800466    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ff5f9f-91c5-4927-910c-974c795bd13b-trusted-ca-bundle\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.800608 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.800340    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24ff5f9f-91c5-4927-910c-974c795bd13b-service-ca\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.802582 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.802542    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24ff5f9f-91c5-4927-910c-974c795bd13b-console-oauth-config\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.802790 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.802771    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24ff5f9f-91c5-4927-910c-974c795bd13b-console-serving-cert\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.811664 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.811643    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwpr6\" (UniqueName: \"kubernetes.io/projected/24ff5f9f-91c5-4927-910c-974c795bd13b-kube-api-access-jwpr6\") pod \"console-677fdbbb87-4h64j\" (UID: \"24ff5f9f-91c5-4927-910c-974c795bd13b\") " pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:39.857879 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:39.857835    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:40.148628 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:40.148591    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-677fdbbb87-4h64j"]
Sep 17 20:56:40.150053 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:56:40.150026    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ff5f9f_91c5_4927_910c_974c795bd13b.slice/crio-2215740e0cd3aff8c306b795e158bf3f6050d4e9fd5364d4b33deb9293382018 WatchSource:0}: Error finding container 2215740e0cd3aff8c306b795e158bf3f6050d4e9fd5364d4b33deb9293382018: Status 404 returned error can't find the container with id 2215740e0cd3aff8c306b795e158bf3f6050d4e9fd5364d4b33deb9293382018
Sep 17 20:56:40.537478 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:40.537394    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-677fdbbb87-4h64j" event={"ID":"24ff5f9f-91c5-4927-910c-974c795bd13b","Type":"ContainerStarted","Data":"6acd3af7d667d1d20adf70c3f1ed3dedbae987c7359fb3afca397c6feae15535"}
Sep 17 20:56:40.537478 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:40.537443    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-677fdbbb87-4h64j" event={"ID":"24ff5f9f-91c5-4927-910c-974c795bd13b","Type":"ContainerStarted","Data":"2215740e0cd3aff8c306b795e158bf3f6050d4e9fd5364d4b33deb9293382018"}
Sep 17 20:56:40.563531 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:40.563476    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-677fdbbb87-4h64j" podStartSLOduration=1.563459736 podStartE2EDuration="1.563459736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:56:40.56262567 +0000 UTC m=+412.067103268" watchObservedRunningTime="2026-09-17 20:56:40.563459736 +0000 UTC m=+412.067937332"
Sep 17 20:56:49.858291 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:49.858240    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:49.858728 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:49.858317    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:49.863417 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:49.863394    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:50.590567 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:50.590520    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-677fdbbb87-4h64j"
Sep 17 20:56:50.640989 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:50.640951    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b776ccfdf-xgd9c"]
Sep 17 20:56:52.637231 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.637197    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-pk4h8"]
Sep 17 20:56:52.639420 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.639402    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8"
Sep 17 20:56:52.643841 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.643810    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 20:56:52.643841 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.643837    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 20:56:52.644008 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.643859    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 20:56:52.644773 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.644759    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-p94zf\""
Sep 17 20:56:52.652859 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.652833    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-pk4h8"]
Sep 17 20:56:52.682051 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.682014    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-zsdnx"]
Sep 17 20:56:52.684158 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.684136    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-zsdnx"
Sep 17 20:56:52.689435 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.689412    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 20:56:52.689569 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.689502    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d2mrz\""
Sep 17 20:56:52.700282 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.700254    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-zsdnx"]
Sep 17 20:56:52.810564 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.810525    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc24k\" (UniqueName: \"kubernetes.io/projected/d53ab2e6-bf64-43ae-afe7-31bb00121e4a-kube-api-access-bc24k\") pod \"seaweedfs-64568bcd49-zsdnx\" (UID: \"d53ab2e6-bf64-43ae-afe7-31bb00121e4a\") " pod="kserve/seaweedfs-64568bcd49-zsdnx"
Sep 17 20:56:52.810755 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.810603    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7xd\" (UniqueName: \"kubernetes.io/projected/87fd10c4-523c-472b-b506-9981e6afcf3f-kube-api-access-qp7xd\") pod \"kserve-controller-manager-66f4b8957f-pk4h8\" (UID: \"87fd10c4-523c-472b-b506-9981e6afcf3f\") " pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8"
Sep 17 20:56:52.810755 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.810645    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d53ab2e6-bf64-43ae-afe7-31bb00121e4a-data\") pod \"seaweedfs-64568bcd49-zsdnx\" (UID: \"d53ab2e6-bf64-43ae-afe7-31bb00121e4a\") " pod="kserve/seaweedfs-64568bcd49-zsdnx"
Sep 17 20:56:52.810755 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.810668    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87fd10c4-523c-472b-b506-9981e6afcf3f-cert\") pod \"kserve-controller-manager-66f4b8957f-pk4h8\" (UID: \"87fd10c4-523c-472b-b506-9981e6afcf3f\") " pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8"
Sep 17 20:56:52.912164 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.912062    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d53ab2e6-bf64-43ae-afe7-31bb00121e4a-data\") pod \"seaweedfs-64568bcd49-zsdnx\" (UID: \"d53ab2e6-bf64-43ae-afe7-31bb00121e4a\") " pod="kserve/seaweedfs-64568bcd49-zsdnx"
Sep 17 20:56:52.912164 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.912116    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87fd10c4-523c-472b-b506-9981e6afcf3f-cert\") pod \"kserve-controller-manager-66f4b8957f-pk4h8\" (UID: \"87fd10c4-523c-472b-b506-9981e6afcf3f\") " pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8"
Sep 17 20:56:52.912400 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.912300    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc24k\" (UniqueName: \"kubernetes.io/projected/d53ab2e6-bf64-43ae-afe7-31bb00121e4a-kube-api-access-bc24k\") pod \"seaweedfs-64568bcd49-zsdnx\" (UID: \"d53ab2e6-bf64-43ae-afe7-31bb00121e4a\") " pod="kserve/seaweedfs-64568bcd49-zsdnx"
Sep 17 20:56:52.912674 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.912649    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7xd\" (UniqueName: \"kubernetes.io/projected/87fd10c4-523c-472b-b506-9981e6afcf3f-kube-api-access-qp7xd\") pod \"kserve-controller-manager-66f4b8957f-pk4h8\" (UID: \"87fd10c4-523c-472b-b506-9981e6afcf3f\") " pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8"
Sep 17 20:56:52.918707 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.913040    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d53ab2e6-bf64-43ae-afe7-31bb00121e4a-data\") pod \"seaweedfs-64568bcd49-zsdnx\" (UID: \"d53ab2e6-bf64-43ae-afe7-31bb00121e4a\") " pod="kserve/seaweedfs-64568bcd49-zsdnx"
Sep 17 20:56:52.919694 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.919668    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87fd10c4-523c-472b-b506-9981e6afcf3f-cert\") pod \"kserve-controller-manager-66f4b8957f-pk4h8\" (UID: \"87fd10c4-523c-472b-b506-9981e6afcf3f\") " pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8"
Sep 17 20:56:52.924697 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.924672    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7xd\" (UniqueName: \"kubernetes.io/projected/87fd10c4-523c-472b-b506-9981e6afcf3f-kube-api-access-qp7xd\") pod \"kserve-controller-manager-66f4b8957f-pk4h8\" (UID: \"87fd10c4-523c-472b-b506-9981e6afcf3f\") " pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8"
Sep 17 20:56:52.926353 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.926321    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc24k\" (UniqueName: \"kubernetes.io/projected/d53ab2e6-bf64-43ae-afe7-31bb00121e4a-kube-api-access-bc24k\") pod \"seaweedfs-64568bcd49-zsdnx\" (UID: \"d53ab2e6-bf64-43ae-afe7-31bb00121e4a\") " pod="kserve/seaweedfs-64568bcd49-zsdnx"
Sep 17 20:56:52.950406 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.950367    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8"
Sep 17 20:56:52.993631 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:52.993591    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-zsdnx"
Sep 17 20:56:53.220745 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:53.220716    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-pk4h8"]
Sep 17 20:56:53.223088 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:56:53.223060    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87fd10c4_523c_472b_b506_9981e6afcf3f.slice/crio-d21bb6ff21b1adb10771cbc846a5920f3d7cbe52701ca2e7bc0844b2fbd89d33 WatchSource:0}: Error finding container d21bb6ff21b1adb10771cbc846a5920f3d7cbe52701ca2e7bc0844b2fbd89d33: Status 404 returned error can't find the container with id d21bb6ff21b1adb10771cbc846a5920f3d7cbe52701ca2e7bc0844b2fbd89d33
Sep 17 20:56:53.277626 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:53.277602    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-zsdnx"]
Sep 17 20:56:53.280190 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:56:53.280167    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53ab2e6_bf64_43ae_afe7_31bb00121e4a.slice/crio-9ea6ab8915bfc08398de7de44fe0e85acd2e321102b88050736b778b1c2b0985 WatchSource:0}: Error finding container 9ea6ab8915bfc08398de7de44fe0e85acd2e321102b88050736b778b1c2b0985: Status 404 returned error can't find the container with id 9ea6ab8915bfc08398de7de44fe0e85acd2e321102b88050736b778b1c2b0985
Sep 17 20:56:53.602536 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:53.602491    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8" event={"ID":"87fd10c4-523c-472b-b506-9981e6afcf3f","Type":"ContainerStarted","Data":"d21bb6ff21b1adb10771cbc846a5920f3d7cbe52701ca2e7bc0844b2fbd89d33"}
Sep 17 20:56:53.603542 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:53.603518    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-zsdnx" event={"ID":"d53ab2e6-bf64-43ae-afe7-31bb00121e4a","Type":"ContainerStarted","Data":"9ea6ab8915bfc08398de7de44fe0e85acd2e321102b88050736b778b1c2b0985"}
Sep 17 20:56:57.627976 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:57.627937    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-zsdnx" event={"ID":"d53ab2e6-bf64-43ae-afe7-31bb00121e4a","Type":"ContainerStarted","Data":"4a0abfb765dca69f9ecc65126f6b745bed4569ed0d1e921c6ae1726073d36dd6"}
Sep 17 20:56:57.628442 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:57.628004    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-zsdnx"
Sep 17 20:56:57.629602 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:57.629573    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8" event={"ID":"87fd10c4-523c-472b-b506-9981e6afcf3f","Type":"ContainerStarted","Data":"2f5624fc3562608757989cb861db9e7e23ef041f81f3c70811d3494a2f7ede4d"}
Sep 17 20:56:57.629708 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:57.629692    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8"
Sep 17 20:56:57.648042 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:57.647997    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-zsdnx" podStartSLOduration=1.737696254 podStartE2EDuration="5.647983324s" totalImagesPullingTime="3.91028707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:56:57.646660761 +0000 UTC m=+429.151138384" watchObservedRunningTime="2026-09-17 20:56:57.647983324 +0000 UTC m=+429.152460953"
Sep 17 20:56:57.663601 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:56:57.663526    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8" podStartSLOduration=1.784797153 podStartE2EDuration="5.663508334s" totalImagesPullingTime="3.878711181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:56:57.663274311 +0000 UTC m=+429.167751908" watchObservedRunningTime="2026-09-17 20:56:57.663508334 +0000 UTC m=+429.167985931"
Sep 17 20:57:03.636372 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:03.636291    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-zsdnx"
Sep 17 20:57:15.662646 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:15.662574    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b776ccfdf-xgd9c" podUID="5e377f08-25ce-4700-b5bf-494d1acbf0a6" containerName="console" containerID="cri-o://9de00bb15e1a9997068f8d6f9a7a1f918c7e572fe8e3dae0a7765ae9809be4be" gracePeriod=15
Sep 17 20:57:15.697634 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:15.697607    2571 patch_prober.go:28] interesting pod/console-5b776ccfdf-xgd9c container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.27:8443/health\": dial tcp 10.134.0.27:8443: connect: connection refused" start-of-body=
Sep 17 20:57:15.697768 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:15.697672    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-5b776ccfdf-xgd9c" podUID="5e377f08-25ce-4700-b5bf-494d1acbf0a6" containerName="console" probeResult="failure" output="Get \"https://10.134.0.27:8443/health\": dial tcp 10.134.0.27:8443: connect: connection refused"
Sep 17 20:57:15.902703 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:15.902678    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:57:16.018043 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.017962    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-trusted-ca-bundle\") pod \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") "
Sep 17 20:57:16.018043 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.018002    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-service-ca\") pod \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") "
Sep 17 20:57:16.018278 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.018055    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-config\") pod \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") "
Sep 17 20:57:16.018278 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.018093    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-oauth-serving-cert\") pod \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") "
Sep 17 20:57:16.018278 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.018128    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-serving-cert\") pod \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") "
Sep 17 20:57:16.018278 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.018214    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-oauth-config\") pod \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") "
Sep 17 20:57:16.018278 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.018246    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gnpp\" (UniqueName: \"kubernetes.io/projected/5e377f08-25ce-4700-b5bf-494d1acbf0a6-kube-api-access-9gnpp\") pod \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\" (UID: \"5e377f08-25ce-4700-b5bf-494d1acbf0a6\") "
Sep 17 20:57:16.018481 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.018445    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5e377f08-25ce-4700-b5bf-494d1acbf0a6" (UID: "5e377f08-25ce-4700-b5bf-494d1acbf0a6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:57:16.018606 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.018577    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5e377f08-25ce-4700-b5bf-494d1acbf0a6" (UID: "5e377f08-25ce-4700-b5bf-494d1acbf0a6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:57:16.018659 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.018603    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-trusted-ca-bundle\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:57:16.018659 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.018586    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-service-ca" (OuterVolumeSpecName: "service-ca") pod "5e377f08-25ce-4700-b5bf-494d1acbf0a6" (UID: "5e377f08-25ce-4700-b5bf-494d1acbf0a6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:57:16.018730 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.018650    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-config" (OuterVolumeSpecName: "console-config") pod "5e377f08-25ce-4700-b5bf-494d1acbf0a6" (UID: "5e377f08-25ce-4700-b5bf-494d1acbf0a6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:57:16.020517 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.020492    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5e377f08-25ce-4700-b5bf-494d1acbf0a6" (UID: "5e377f08-25ce-4700-b5bf-494d1acbf0a6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:57:16.020737 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.020719    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e377f08-25ce-4700-b5bf-494d1acbf0a6-kube-api-access-9gnpp" (OuterVolumeSpecName: "kube-api-access-9gnpp") pod "5e377f08-25ce-4700-b5bf-494d1acbf0a6" (UID: "5e377f08-25ce-4700-b5bf-494d1acbf0a6"). InnerVolumeSpecName "kube-api-access-9gnpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:57:16.020737 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.020722    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5e377f08-25ce-4700-b5bf-494d1acbf0a6" (UID: "5e377f08-25ce-4700-b5bf-494d1acbf0a6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:57:16.119062 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.119025    2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-service-ca\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:57:16.119062 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.119057    2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:57:16.119062 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.119067    2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e377f08-25ce-4700-b5bf-494d1acbf0a6-oauth-serving-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:57:16.119062 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.119075    2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-serving-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:57:16.119323 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.119085    2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e377f08-25ce-4700-b5bf-494d1acbf0a6-console-oauth-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:57:16.119323 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.119095    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gnpp\" (UniqueName: \"kubernetes.io/projected/5e377f08-25ce-4700-b5bf-494d1acbf0a6-kube-api-access-9gnpp\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:57:16.716295 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.716258    2571 generic.go:356] "Generic (PLEG): container finished" podID="5e377f08-25ce-4700-b5bf-494d1acbf0a6" containerID="9de00bb15e1a9997068f8d6f9a7a1f918c7e572fe8e3dae0a7765ae9809be4be" exitCode=0
Sep 17 20:57:16.716700 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.716309    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b776ccfdf-xgd9c"
Sep 17 20:57:16.716700 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.716353    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b776ccfdf-xgd9c" event={"ID":"5e377f08-25ce-4700-b5bf-494d1acbf0a6","Type":"ContainerDied","Data":"9de00bb15e1a9997068f8d6f9a7a1f918c7e572fe8e3dae0a7765ae9809be4be"}
Sep 17 20:57:16.716700 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.716407    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b776ccfdf-xgd9c" event={"ID":"5e377f08-25ce-4700-b5bf-494d1acbf0a6","Type":"ContainerDied","Data":"d27469a81dcb1ac69e1af9c738e71d4d9fa861fb5cb4c46b87b4f4e0b4a58603"}
Sep 17 20:57:16.716700 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.716433    2571 scope.go:117] "RemoveContainer" containerID="9de00bb15e1a9997068f8d6f9a7a1f918c7e572fe8e3dae0a7765ae9809be4be"
Sep 17 20:57:16.725017 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.724998    2571 scope.go:117] "RemoveContainer" containerID="9de00bb15e1a9997068f8d6f9a7a1f918c7e572fe8e3dae0a7765ae9809be4be"
Sep 17 20:57:16.725287 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:57:16.725272    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de00bb15e1a9997068f8d6f9a7a1f918c7e572fe8e3dae0a7765ae9809be4be\": container with ID starting with 9de00bb15e1a9997068f8d6f9a7a1f918c7e572fe8e3dae0a7765ae9809be4be not found: ID does not exist" containerID="9de00bb15e1a9997068f8d6f9a7a1f918c7e572fe8e3dae0a7765ae9809be4be"
Sep 17 20:57:16.725333 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.725296    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de00bb15e1a9997068f8d6f9a7a1f918c7e572fe8e3dae0a7765ae9809be4be"} err="failed to get container status \"9de00bb15e1a9997068f8d6f9a7a1f918c7e572fe8e3dae0a7765ae9809be4be\": rpc error: code = NotFound desc = could not find container \"9de00bb15e1a9997068f8d6f9a7a1f918c7e572fe8e3dae0a7765ae9809be4be\": container with ID starting with 9de00bb15e1a9997068f8d6f9a7a1f918c7e572fe8e3dae0a7765ae9809be4be not found: ID does not exist"
Sep 17 20:57:16.739598 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.739573    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b776ccfdf-xgd9c"]
Sep 17 20:57:16.743799 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:16.743776    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b776ccfdf-xgd9c"]
Sep 17 20:57:18.670266 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:18.670234    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e377f08-25ce-4700-b5bf-494d1acbf0a6" path="/var/lib/kubelet/pods/5e377f08-25ce-4700-b5bf-494d1acbf0a6/volumes"
Sep 17 20:57:28.639917 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:28.639887    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8"
Sep 17 20:57:39.985035 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:39.984996    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-pk4h8"]
Sep 17 20:57:39.985462 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:39.985209    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8" podUID="87fd10c4-523c-472b-b506-9981e6afcf3f" containerName="manager" containerID="cri-o://2f5624fc3562608757989cb861db9e7e23ef041f81f3c70811d3494a2f7ede4d" gracePeriod=10
Sep 17 20:57:40.049631 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.049597    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-csj4z"]
Sep 17 20:57:40.049955 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.049942    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e377f08-25ce-4700-b5bf-494d1acbf0a6" containerName="console"
Sep 17 20:57:40.049998 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.049957    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e377f08-25ce-4700-b5bf-494d1acbf0a6" containerName="console"
Sep 17 20:57:40.050032 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.050019    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e377f08-25ce-4700-b5bf-494d1acbf0a6" containerName="console"
Sep 17 20:57:40.051749 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.051735    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-csj4z"
Sep 17 20:57:40.083263 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.083235    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-csj4z"]
Sep 17 20:57:40.123111 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.123081    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c297151d-b138-4f59-910c-72f69ae716e6-cert\") pod \"kserve-controller-manager-66f4b8957f-csj4z\" (UID: \"c297151d-b138-4f59-910c-72f69ae716e6\") " pod="kserve/kserve-controller-manager-66f4b8957f-csj4z"
Sep 17 20:57:40.123268 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.123124    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp7hl\" (UniqueName: \"kubernetes.io/projected/c297151d-b138-4f59-910c-72f69ae716e6-kube-api-access-hp7hl\") pod \"kserve-controller-manager-66f4b8957f-csj4z\" (UID: \"c297151d-b138-4f59-910c-72f69ae716e6\") " pod="kserve/kserve-controller-manager-66f4b8957f-csj4z"
Sep 17 20:57:40.224053 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.224023    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp7hl\" (UniqueName: \"kubernetes.io/projected/c297151d-b138-4f59-910c-72f69ae716e6-kube-api-access-hp7hl\") pod \"kserve-controller-manager-66f4b8957f-csj4z\" (UID: \"c297151d-b138-4f59-910c-72f69ae716e6\") " pod="kserve/kserve-controller-manager-66f4b8957f-csj4z"
Sep 17 20:57:40.224203 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.224109    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c297151d-b138-4f59-910c-72f69ae716e6-cert\") pod \"kserve-controller-manager-66f4b8957f-csj4z\" (UID: \"c297151d-b138-4f59-910c-72f69ae716e6\") " pod="kserve/kserve-controller-manager-66f4b8957f-csj4z"
Sep 17 20:57:40.226525 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.226502    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c297151d-b138-4f59-910c-72f69ae716e6-cert\") pod \"kserve-controller-manager-66f4b8957f-csj4z\" (UID: \"c297151d-b138-4f59-910c-72f69ae716e6\") " pod="kserve/kserve-controller-manager-66f4b8957f-csj4z"
Sep 17 20:57:40.235270 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.235217    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp7hl\" (UniqueName: \"kubernetes.io/projected/c297151d-b138-4f59-910c-72f69ae716e6-kube-api-access-hp7hl\") pod \"kserve-controller-manager-66f4b8957f-csj4z\" (UID: \"c297151d-b138-4f59-910c-72f69ae716e6\") " pod="kserve/kserve-controller-manager-66f4b8957f-csj4z"
Sep 17 20:57:40.247632 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.247606    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8"
Sep 17 20:57:40.325380 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.325350    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp7xd\" (UniqueName: \"kubernetes.io/projected/87fd10c4-523c-472b-b506-9981e6afcf3f-kube-api-access-qp7xd\") pod \"87fd10c4-523c-472b-b506-9981e6afcf3f\" (UID: \"87fd10c4-523c-472b-b506-9981e6afcf3f\") "
Sep 17 20:57:40.325380 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.325387    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87fd10c4-523c-472b-b506-9981e6afcf3f-cert\") pod \"87fd10c4-523c-472b-b506-9981e6afcf3f\" (UID: \"87fd10c4-523c-472b-b506-9981e6afcf3f\") "
Sep 17 20:57:40.327823 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.327784    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87fd10c4-523c-472b-b506-9981e6afcf3f-cert" (OuterVolumeSpecName: "cert") pod "87fd10c4-523c-472b-b506-9981e6afcf3f" (UID: "87fd10c4-523c-472b-b506-9981e6afcf3f"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:57:40.327936 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.327898    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87fd10c4-523c-472b-b506-9981e6afcf3f-kube-api-access-qp7xd" (OuterVolumeSpecName: "kube-api-access-qp7xd") pod "87fd10c4-523c-472b-b506-9981e6afcf3f" (UID: "87fd10c4-523c-472b-b506-9981e6afcf3f"). InnerVolumeSpecName "kube-api-access-qp7xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:57:40.363696 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.363661    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-csj4z"
Sep 17 20:57:40.426222 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.426177    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp7xd\" (UniqueName: \"kubernetes.io/projected/87fd10c4-523c-472b-b506-9981e6afcf3f-kube-api-access-qp7xd\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:57:40.426222 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.426202    2571 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87fd10c4-523c-472b-b506-9981e6afcf3f-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:57:40.640963 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.640924    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-csj4z"]
Sep 17 20:57:40.646927 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:57:40.646897    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc297151d_b138_4f59_910c_72f69ae716e6.slice/crio-d68cce71fdf5819ef4d0a1992e1785361603ffc442a876bc777cfb3dd08803d8 WatchSource:0}: Error finding container d68cce71fdf5819ef4d0a1992e1785361603ffc442a876bc777cfb3dd08803d8: Status 404 returned error can't find the container with id d68cce71fdf5819ef4d0a1992e1785361603ffc442a876bc777cfb3dd08803d8
Sep 17 20:57:40.829396 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.829359    2571 generic.go:356] "Generic (PLEG): container finished" podID="87fd10c4-523c-472b-b506-9981e6afcf3f" containerID="2f5624fc3562608757989cb861db9e7e23ef041f81f3c70811d3494a2f7ede4d" exitCode=0
Sep 17 20:57:40.829604 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.829407    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8"
Sep 17 20:57:40.829604 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.829441    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8" event={"ID":"87fd10c4-523c-472b-b506-9981e6afcf3f","Type":"ContainerDied","Data":"2f5624fc3562608757989cb861db9e7e23ef041f81f3c70811d3494a2f7ede4d"}
Sep 17 20:57:40.829604 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.829501    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-pk4h8" event={"ID":"87fd10c4-523c-472b-b506-9981e6afcf3f","Type":"ContainerDied","Data":"d21bb6ff21b1adb10771cbc846a5920f3d7cbe52701ca2e7bc0844b2fbd89d33"}
Sep 17 20:57:40.829604 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.829524    2571 scope.go:117] "RemoveContainer" containerID="2f5624fc3562608757989cb861db9e7e23ef041f81f3c70811d3494a2f7ede4d"
Sep 17 20:57:40.831000 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.830976    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-csj4z" event={"ID":"c297151d-b138-4f59-910c-72f69ae716e6","Type":"ContainerStarted","Data":"d68cce71fdf5819ef4d0a1992e1785361603ffc442a876bc777cfb3dd08803d8"}
Sep 17 20:57:40.837818 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.837802    2571 scope.go:117] "RemoveContainer" containerID="2f5624fc3562608757989cb861db9e7e23ef041f81f3c70811d3494a2f7ede4d"
Sep 17 20:57:40.838074 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:57:40.838051    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5624fc3562608757989cb861db9e7e23ef041f81f3c70811d3494a2f7ede4d\": container with ID starting with 2f5624fc3562608757989cb861db9e7e23ef041f81f3c70811d3494a2f7ede4d not found: ID does not exist" containerID="2f5624fc3562608757989cb861db9e7e23ef041f81f3c70811d3494a2f7ede4d"
Sep 17 20:57:40.838130 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.838081    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5624fc3562608757989cb861db9e7e23ef041f81f3c70811d3494a2f7ede4d"} err="failed to get container status \"2f5624fc3562608757989cb861db9e7e23ef041f81f3c70811d3494a2f7ede4d\": rpc error: code = NotFound desc = could not find container \"2f5624fc3562608757989cb861db9e7e23ef041f81f3c70811d3494a2f7ede4d\": container with ID starting with 2f5624fc3562608757989cb861db9e7e23ef041f81f3c70811d3494a2f7ede4d not found: ID does not exist"
Sep 17 20:57:40.872083 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.872059    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-pk4h8"]
Sep 17 20:57:40.883340 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:40.883312    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-66f4b8957f-pk4h8"]
Sep 17 20:57:41.837961 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:41.837922    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f4b8957f-csj4z" event={"ID":"c297151d-b138-4f59-910c-72f69ae716e6","Type":"ContainerStarted","Data":"f6993369c2838efc5a7593933ef8b827d1e039f8bf5b483d2b33b4d92f72872e"}
Sep 17 20:57:41.838429 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:41.837999    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-66f4b8957f-csj4z"
Sep 17 20:57:41.865983 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:41.865878    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-66f4b8957f-csj4z" podStartSLOduration=2.433807313 podStartE2EDuration="2.865859229s" totalImagesPullingTime="432.051916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:57:41.863428221 +0000 UTC m=+473.367905827" watchObservedRunningTime="2026-09-17 20:57:41.865859229 +0000 UTC m=+473.370336831"
Sep 17 20:57:42.672129 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:57:42.672094    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87fd10c4-523c-472b-b506-9981e6afcf3f" path="/var/lib/kubelet/pods/87fd10c4-523c-472b-b506-9981e6afcf3f/volumes"
Sep 17 20:58:12.848883 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:12.848852    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-66f4b8957f-csj4z"
Sep 17 20:58:14.442782 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:14.442751    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-22vrx"]
Sep 17 20:58:14.443134 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:14.443097    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87fd10c4-523c-472b-b506-9981e6afcf3f" containerName="manager"
Sep 17 20:58:14.443134 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:14.443108    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="87fd10c4-523c-472b-b506-9981e6afcf3f" containerName="manager"
Sep 17 20:58:14.443207 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:14.443169    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="87fd10c4-523c-472b-b506-9981e6afcf3f" containerName="manager"
Sep 17 20:58:14.446095 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:14.446081    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-22vrx"
Sep 17 20:58:14.448471 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:14.448447    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 20:58:14.448613 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:14.448479    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-4tnw2\""
Sep 17 20:58:14.457954 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:14.457927    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-22vrx"]
Sep 17 20:58:14.523886 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:14.523847    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/382c2a30-c5dd-42a1-b5ef-d990faa733fa-tls-certs\") pod \"model-serving-api-7b954ffc95-22vrx\" (UID: \"382c2a30-c5dd-42a1-b5ef-d990faa733fa\") " pod="kserve/model-serving-api-7b954ffc95-22vrx"
Sep 17 20:58:14.524058 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:14.523924    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpspq\" (UniqueName: \"kubernetes.io/projected/382c2a30-c5dd-42a1-b5ef-d990faa733fa-kube-api-access-hpspq\") pod \"model-serving-api-7b954ffc95-22vrx\" (UID: \"382c2a30-c5dd-42a1-b5ef-d990faa733fa\") " pod="kserve/model-serving-api-7b954ffc95-22vrx"
Sep 17 20:58:14.625300 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:14.625261    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/382c2a30-c5dd-42a1-b5ef-d990faa733fa-tls-certs\") pod \"model-serving-api-7b954ffc95-22vrx\" (UID: \"382c2a30-c5dd-42a1-b5ef-d990faa733fa\") " pod="kserve/model-serving-api-7b954ffc95-22vrx"
Sep 17 20:58:14.625509 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:14.625341    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpspq\" (UniqueName: \"kubernetes.io/projected/382c2a30-c5dd-42a1-b5ef-d990faa733fa-kube-api-access-hpspq\") pod \"model-serving-api-7b954ffc95-22vrx\" (UID: \"382c2a30-c5dd-42a1-b5ef-d990faa733fa\") " pod="kserve/model-serving-api-7b954ffc95-22vrx"
Sep 17 20:58:14.627886 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:14.627866    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/382c2a30-c5dd-42a1-b5ef-d990faa733fa-tls-certs\") pod \"model-serving-api-7b954ffc95-22vrx\" (UID: \"382c2a30-c5dd-42a1-b5ef-d990faa733fa\") " pod="kserve/model-serving-api-7b954ffc95-22vrx"
Sep 17 20:58:14.634348 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:14.634316    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpspq\" (UniqueName: \"kubernetes.io/projected/382c2a30-c5dd-42a1-b5ef-d990faa733fa-kube-api-access-hpspq\") pod \"model-serving-api-7b954ffc95-22vrx\" (UID: \"382c2a30-c5dd-42a1-b5ef-d990faa733fa\") " pod="kserve/model-serving-api-7b954ffc95-22vrx"
Sep 17 20:58:14.756805 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:14.756713    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-22vrx"
Sep 17 20:58:15.104186 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:15.104155    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-22vrx"]
Sep 17 20:58:15.998434 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:15.998390    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-22vrx" event={"ID":"382c2a30-c5dd-42a1-b5ef-d990faa733fa","Type":"ContainerStarted","Data":"cde4c9ca84ad9ddf2617d9729cfff2afbd7d93f7446f3ef64bcf2d900b682889"}
Sep 17 20:58:18.009819 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:18.009786    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-22vrx" event={"ID":"382c2a30-c5dd-42a1-b5ef-d990faa733fa","Type":"ContainerStarted","Data":"3c2bb0a009f99d4847d3bdc84bb540b0cfefb631ec62f58d507d5776e9ef9411"}
Sep 17 20:58:18.010213 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:18.009916    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-22vrx"
Sep 17 20:58:18.027737 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:18.027689    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-22vrx" podStartSLOduration=1.8262459039999999 podStartE2EDuration="4.02767443s" totalImagesPullingTime="2.201428526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:58:18.02752375 +0000 UTC m=+509.532001338" watchObservedRunningTime="2026-09-17 20:58:18.02767443 +0000 UTC m=+509.532152026"
Sep 17 20:58:29.019313 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:58:29.019281    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-22vrx"
Sep 17 20:59:49.438074 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:49.438038    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc"]
Sep 17 20:59:49.440831 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:49.440810    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc"
Sep 17 20:59:49.443875 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:49.443855    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 20:59:49.448996 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:49.448972    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc"]
Sep 17 20:59:49.545117 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:49.545075    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29cabeb8-fc5a-49eb-b348-b88b0a6efa70-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-9nbgc\" (UID: \"29cabeb8-fc5a-49eb-b348-b88b0a6efa70\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc"
Sep 17 20:59:49.545293 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:49.545137    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd7kd\" (UniqueName: \"kubernetes.io/projected/29cabeb8-fc5a-49eb-b348-b88b0a6efa70-kube-api-access-bd7kd\") pod \"seaweedfs-tls-custom-6644bf5d6c-9nbgc\" (UID: \"29cabeb8-fc5a-49eb-b348-b88b0a6efa70\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc"
Sep 17 20:59:49.646394 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:49.646364    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7kd\" (UniqueName: \"kubernetes.io/projected/29cabeb8-fc5a-49eb-b348-b88b0a6efa70-kube-api-access-bd7kd\") pod \"seaweedfs-tls-custom-6644bf5d6c-9nbgc\" (UID: \"29cabeb8-fc5a-49eb-b348-b88b0a6efa70\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc"
Sep 17 20:59:49.646512 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:49.646457    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29cabeb8-fc5a-49eb-b348-b88b0a6efa70-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-9nbgc\" (UID: \"29cabeb8-fc5a-49eb-b348-b88b0a6efa70\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc"
Sep 17 20:59:49.646891 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:49.646873    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29cabeb8-fc5a-49eb-b348-b88b0a6efa70-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-9nbgc\" (UID: \"29cabeb8-fc5a-49eb-b348-b88b0a6efa70\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc"
Sep 17 20:59:49.655167 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:49.655146    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd7kd\" (UniqueName: \"kubernetes.io/projected/29cabeb8-fc5a-49eb-b348-b88b0a6efa70-kube-api-access-bd7kd\") pod \"seaweedfs-tls-custom-6644bf5d6c-9nbgc\" (UID: \"29cabeb8-fc5a-49eb-b348-b88b0a6efa70\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc"
Sep 17 20:59:49.750518 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:49.750424    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc"
Sep 17 20:59:50.021444 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:50.021415    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc"]
Sep 17 20:59:50.023955 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:59:50.023927    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29cabeb8_fc5a_49eb_b348_b88b0a6efa70.slice/crio-681cdfae7946060f950ef0a545a1f9a6147f7cd07d66331bf7f2f2354a0c14d1 WatchSource:0}: Error finding container 681cdfae7946060f950ef0a545a1f9a6147f7cd07d66331bf7f2f2354a0c14d1: Status 404 returned error can't find the container with id 681cdfae7946060f950ef0a545a1f9a6147f7cd07d66331bf7f2f2354a0c14d1
Sep 17 20:59:50.447483 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:50.447434    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc" event={"ID":"29cabeb8-fc5a-49eb-b348-b88b0a6efa70","Type":"ContainerStarted","Data":"eb28f8ac8906bee84a67b66f759d28b599caed9c8c2a4acf6ed3efebd14a0b1b"}
Sep 17 20:59:50.447883 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:50.447493    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc" event={"ID":"29cabeb8-fc5a-49eb-b348-b88b0a6efa70","Type":"ContainerStarted","Data":"681cdfae7946060f950ef0a545a1f9a6147f7cd07d66331bf7f2f2354a0c14d1"}
Sep 17 20:59:50.463348 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:50.463303    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc" podStartSLOduration=1.168089865 podStartE2EDuration="1.463286441s" totalImagesPullingTime="295.196576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:59:50.463194196 +0000 UTC m=+601.967671793" watchObservedRunningTime="2026-09-17 20:59:50.463286441 +0000 UTC m=+601.967764075"
Sep 17 20:59:51.629939 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:51.629903    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc"]
Sep 17 20:59:52.456701 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:52.456638    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc" podUID="29cabeb8-fc5a-49eb-b348-b88b0a6efa70" containerName="seaweedfs-tls-custom" containerID="cri-o://eb28f8ac8906bee84a67b66f759d28b599caed9c8c2a4acf6ed3efebd14a0b1b" gracePeriod=30
Sep 17 20:59:53.701342 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:53.701315    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc"
Sep 17 20:59:53.781208 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:53.781131    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd7kd\" (UniqueName: \"kubernetes.io/projected/29cabeb8-fc5a-49eb-b348-b88b0a6efa70-kube-api-access-bd7kd\") pod \"29cabeb8-fc5a-49eb-b348-b88b0a6efa70\" (UID: \"29cabeb8-fc5a-49eb-b348-b88b0a6efa70\") "
Sep 17 20:59:53.781208 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:53.781189    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29cabeb8-fc5a-49eb-b348-b88b0a6efa70-data\") pod \"29cabeb8-fc5a-49eb-b348-b88b0a6efa70\" (UID: \"29cabeb8-fc5a-49eb-b348-b88b0a6efa70\") "
Sep 17 20:59:53.782823 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:53.782798    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cabeb8-fc5a-49eb-b348-b88b0a6efa70-data" (OuterVolumeSpecName: "data") pod "29cabeb8-fc5a-49eb-b348-b88b0a6efa70" (UID: "29cabeb8-fc5a-49eb-b348-b88b0a6efa70"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:59:53.783456 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:53.783430    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29cabeb8-fc5a-49eb-b348-b88b0a6efa70-kube-api-access-bd7kd" (OuterVolumeSpecName: "kube-api-access-bd7kd") pod "29cabeb8-fc5a-49eb-b348-b88b0a6efa70" (UID: "29cabeb8-fc5a-49eb-b348-b88b0a6efa70"). InnerVolumeSpecName "kube-api-access-bd7kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:59:53.882680 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:53.882625    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd7kd\" (UniqueName: \"kubernetes.io/projected/29cabeb8-fc5a-49eb-b348-b88b0a6efa70-kube-api-access-bd7kd\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:59:53.882680 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:53.882670    2571 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29cabeb8-fc5a-49eb-b348-b88b0a6efa70-data\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 20:59:54.468173 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.468140    2571 generic.go:356] "Generic (PLEG): container finished" podID="29cabeb8-fc5a-49eb-b348-b88b0a6efa70" containerID="eb28f8ac8906bee84a67b66f759d28b599caed9c8c2a4acf6ed3efebd14a0b1b" exitCode=0
Sep 17 20:59:54.468337 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.468185    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc"
Sep 17 20:59:54.468337 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.468226    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc" event={"ID":"29cabeb8-fc5a-49eb-b348-b88b0a6efa70","Type":"ContainerDied","Data":"eb28f8ac8906bee84a67b66f759d28b599caed9c8c2a4acf6ed3efebd14a0b1b"}
Sep 17 20:59:54.468337 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.468276    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc" event={"ID":"29cabeb8-fc5a-49eb-b348-b88b0a6efa70","Type":"ContainerDied","Data":"681cdfae7946060f950ef0a545a1f9a6147f7cd07d66331bf7f2f2354a0c14d1"}
Sep 17 20:59:54.468337 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.468297    2571 scope.go:117] "RemoveContainer" containerID="eb28f8ac8906bee84a67b66f759d28b599caed9c8c2a4acf6ed3efebd14a0b1b"
Sep 17 20:59:54.478123 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.478101    2571 scope.go:117] "RemoveContainer" containerID="eb28f8ac8906bee84a67b66f759d28b599caed9c8c2a4acf6ed3efebd14a0b1b"
Sep 17 20:59:54.478434 ip-10-0-131-94 kubenswrapper[2571]: E0917 20:59:54.478406    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb28f8ac8906bee84a67b66f759d28b599caed9c8c2a4acf6ed3efebd14a0b1b\": container with ID starting with eb28f8ac8906bee84a67b66f759d28b599caed9c8c2a4acf6ed3efebd14a0b1b not found: ID does not exist" containerID="eb28f8ac8906bee84a67b66f759d28b599caed9c8c2a4acf6ed3efebd14a0b1b"
Sep 17 20:59:54.478522 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.478444    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb28f8ac8906bee84a67b66f759d28b599caed9c8c2a4acf6ed3efebd14a0b1b"} err="failed to get container status \"eb28f8ac8906bee84a67b66f759d28b599caed9c8c2a4acf6ed3efebd14a0b1b\": rpc error: code = NotFound desc = could not find container \"eb28f8ac8906bee84a67b66f759d28b599caed9c8c2a4acf6ed3efebd14a0b1b\": container with ID starting with eb28f8ac8906bee84a67b66f759d28b599caed9c8c2a4acf6ed3efebd14a0b1b not found: ID does not exist"
Sep 17 20:59:54.489586 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.489566    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc"]
Sep 17 20:59:54.493742 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.493720    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9nbgc"]
Sep 17 20:59:54.525170 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.525127    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-k9mr6"]
Sep 17 20:59:54.525970 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.525944    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29cabeb8-fc5a-49eb-b348-b88b0a6efa70" containerName="seaweedfs-tls-custom"
Sep 17 20:59:54.525970 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.525971    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="29cabeb8-fc5a-49eb-b348-b88b0a6efa70" containerName="seaweedfs-tls-custom"
Sep 17 20:59:54.526156 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.526149    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="29cabeb8-fc5a-49eb-b348-b88b0a6efa70" containerName="seaweedfs-tls-custom"
Sep 17 20:59:54.532146 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.532118    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-k9mr6"
Sep 17 20:59:54.533607 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.533586    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-k9mr6"]
Sep 17 20:59:54.534992 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.534973    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 20:59:54.535095 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.535077    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 17 20:59:54.590806 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.590774    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9f833c5f-aebb-478d-ab54-abab238ee436-data\") pod \"seaweedfs-tls-custom-8b44d767d-k9mr6\" (UID: \"9f833c5f-aebb-478d-ab54-abab238ee436\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k9mr6"
Sep 17 20:59:54.590989 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.590841    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/9f833c5f-aebb-478d-ab54-abab238ee436-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-k9mr6\" (UID: \"9f833c5f-aebb-478d-ab54-abab238ee436\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k9mr6"
Sep 17 20:59:54.590989 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.590926    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt57f\" (UniqueName: \"kubernetes.io/projected/9f833c5f-aebb-478d-ab54-abab238ee436-kube-api-access-jt57f\") pod \"seaweedfs-tls-custom-8b44d767d-k9mr6\" (UID: \"9f833c5f-aebb-478d-ab54-abab238ee436\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k9mr6"
Sep 17 20:59:54.669903 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.669859    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29cabeb8-fc5a-49eb-b348-b88b0a6efa70" path="/var/lib/kubelet/pods/29cabeb8-fc5a-49eb-b348-b88b0a6efa70/volumes"
Sep 17 20:59:54.691349 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.691325    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt57f\" (UniqueName: \"kubernetes.io/projected/9f833c5f-aebb-478d-ab54-abab238ee436-kube-api-access-jt57f\") pod \"seaweedfs-tls-custom-8b44d767d-k9mr6\" (UID: \"9f833c5f-aebb-478d-ab54-abab238ee436\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k9mr6"
Sep 17 20:59:54.691449 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.691361    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9f833c5f-aebb-478d-ab54-abab238ee436-data\") pod \"seaweedfs-tls-custom-8b44d767d-k9mr6\" (UID: \"9f833c5f-aebb-478d-ab54-abab238ee436\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k9mr6"
Sep 17 20:59:54.691449 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.691420    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/9f833c5f-aebb-478d-ab54-abab238ee436-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-k9mr6\" (UID: \"9f833c5f-aebb-478d-ab54-abab238ee436\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k9mr6"
Sep 17 20:59:54.691806 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.691787    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9f833c5f-aebb-478d-ab54-abab238ee436-data\") pod \"seaweedfs-tls-custom-8b44d767d-k9mr6\" (UID: \"9f833c5f-aebb-478d-ab54-abab238ee436\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k9mr6"
Sep 17 20:59:54.693978 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.693961    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/9f833c5f-aebb-478d-ab54-abab238ee436-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-k9mr6\" (UID: \"9f833c5f-aebb-478d-ab54-abab238ee436\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k9mr6"
Sep 17 20:59:54.700370 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.700351    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt57f\" (UniqueName: \"kubernetes.io/projected/9f833c5f-aebb-478d-ab54-abab238ee436-kube-api-access-jt57f\") pod \"seaweedfs-tls-custom-8b44d767d-k9mr6\" (UID: \"9f833c5f-aebb-478d-ab54-abab238ee436\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-k9mr6"
Sep 17 20:59:54.841906 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:54.841877    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-k9mr6"
Sep 17 20:59:55.110945 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:55.110918    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-k9mr6"]
Sep 17 20:59:55.113241 ip-10-0-131-94 kubenswrapper[2571]: W0917 20:59:55.113206    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f833c5f_aebb_478d_ab54_abab238ee436.slice/crio-caa25931e0fed4451c8bd14fe6c64de9699a3d892582eca30a9d683eaae1f7f1 WatchSource:0}: Error finding container caa25931e0fed4451c8bd14fe6c64de9699a3d892582eca30a9d683eaae1f7f1: Status 404 returned error can't find the container with id caa25931e0fed4451c8bd14fe6c64de9699a3d892582eca30a9d683eaae1f7f1
Sep 17 20:59:55.474692 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:55.474652    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-k9mr6" event={"ID":"9f833c5f-aebb-478d-ab54-abab238ee436","Type":"ContainerStarted","Data":"580749d86f15d6c2369deb31d76516d07dc916a77997c8b760473fbd13509d80"}
Sep 17 20:59:55.474875 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:55.474710    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-k9mr6" event={"ID":"9f833c5f-aebb-478d-ab54-abab238ee436","Type":"ContainerStarted","Data":"caa25931e0fed4451c8bd14fe6c64de9699a3d892582eca30a9d683eaae1f7f1"}
Sep 17 20:59:55.493626 ip-10-0-131-94 kubenswrapper[2571]: I0917 20:59:55.493569    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-k9mr6" podStartSLOduration=1.251878464 podStartE2EDuration="1.493538856s" totalImagesPullingTime="241.660392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:59:55.491596688 +0000 UTC m=+606.996074336" watchObservedRunningTime="2026-09-17 20:59:55.493538856 +0000 UTC m=+606.998016452"
Sep 17 21:00:08.382884 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.382846    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g"]
Sep 17 21:00:08.386422 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.386400    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g"
Sep 17 21:00:08.389029 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.389010    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 17 21:00:08.389111 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.389015    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 21:00:08.394539 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.394501    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g"]
Sep 17 21:00:08.517714 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.517660    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98bfs\" (UniqueName: \"kubernetes.io/projected/ada3840e-1c1c-47e8-81c6-d73f67164067-kube-api-access-98bfs\") pod \"seaweedfs-tls-serving-67b7477c6d-zvd7g\" (UID: \"ada3840e-1c1c-47e8-81c6-d73f67164067\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g"
Sep 17 21:00:08.517926 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.517737    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/ada3840e-1c1c-47e8-81c6-d73f67164067-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-zvd7g\" (UID: \"ada3840e-1c1c-47e8-81c6-d73f67164067\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g"
Sep 17 21:00:08.517926 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.517766    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ada3840e-1c1c-47e8-81c6-d73f67164067-data\") pod \"seaweedfs-tls-serving-67b7477c6d-zvd7g\" (UID: \"ada3840e-1c1c-47e8-81c6-d73f67164067\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g"
Sep 17 21:00:08.618982 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.618945    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bfs\" (UniqueName: \"kubernetes.io/projected/ada3840e-1c1c-47e8-81c6-d73f67164067-kube-api-access-98bfs\") pod \"seaweedfs-tls-serving-67b7477c6d-zvd7g\" (UID: \"ada3840e-1c1c-47e8-81c6-d73f67164067\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g"
Sep 17 21:00:08.619157 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.619001    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/ada3840e-1c1c-47e8-81c6-d73f67164067-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-zvd7g\" (UID: \"ada3840e-1c1c-47e8-81c6-d73f67164067\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g"
Sep 17 21:00:08.619157 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.619032    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ada3840e-1c1c-47e8-81c6-d73f67164067-data\") pod \"seaweedfs-tls-serving-67b7477c6d-zvd7g\" (UID: \"ada3840e-1c1c-47e8-81c6-d73f67164067\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g"
Sep 17 21:00:08.619588 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.619528    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ada3840e-1c1c-47e8-81c6-d73f67164067-data\") pod \"seaweedfs-tls-serving-67b7477c6d-zvd7g\" (UID: \"ada3840e-1c1c-47e8-81c6-d73f67164067\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g"
Sep 17 21:00:08.621658 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.621636    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/ada3840e-1c1c-47e8-81c6-d73f67164067-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-zvd7g\" (UID: \"ada3840e-1c1c-47e8-81c6-d73f67164067\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g"
Sep 17 21:00:08.629066 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.629041    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98bfs\" (UniqueName: \"kubernetes.io/projected/ada3840e-1c1c-47e8-81c6-d73f67164067-kube-api-access-98bfs\") pod \"seaweedfs-tls-serving-67b7477c6d-zvd7g\" (UID: \"ada3840e-1c1c-47e8-81c6-d73f67164067\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g"
Sep 17 21:00:08.697478 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.697389    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g"
Sep 17 21:00:08.969361 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.969335    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g"]
Sep 17 21:00:08.971914 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:00:08.971884    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada3840e_1c1c_47e8_81c6_d73f67164067.slice/crio-4009dea0040be97851c5eab45d19a15d3db710c4e92bc04d56de9b4b4c1a6b01 WatchSource:0}: Error finding container 4009dea0040be97851c5eab45d19a15d3db710c4e92bc04d56de9b4b4c1a6b01: Status 404 returned error can't find the container with id 4009dea0040be97851c5eab45d19a15d3db710c4e92bc04d56de9b4b4c1a6b01
Sep 17 21:00:08.973247 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:08.973225    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:00:09.545054 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:09.544969    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g" event={"ID":"ada3840e-1c1c-47e8-81c6-d73f67164067","Type":"ContainerStarted","Data":"b7566dd641f9cf505c00c0ba67ce1b20717764ad1813de10986eb17548cefff2"}
Sep 17 21:00:09.545054 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:09.545018    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g" event={"ID":"ada3840e-1c1c-47e8-81c6-d73f67164067","Type":"ContainerStarted","Data":"4009dea0040be97851c5eab45d19a15d3db710c4e92bc04d56de9b4b4c1a6b01"}
Sep 17 21:00:09.564456 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:00:09.564384    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-zvd7g" podStartSLOduration=1.28772391 podStartE2EDuration="1.564364528s" totalImagesPullingTime="276.640618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:00:09.562988858 +0000 UTC m=+621.067466478" watchObservedRunningTime="2026-09-17 21:00:09.564364528 +0000 UTC m=+621.068842179"
Sep 17 21:06:06.413624 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.413523    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"]
Sep 17 21:06:06.417241 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.417225    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:06.421051 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.421027    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:06:06.421211 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.421070    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 17 21:06:06.421211 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.421167    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:06:06.421390 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.421287    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:06:06.421450 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.421398    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 21:06:06.421450 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.421441    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:06:06.422612 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.422591    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:06:06.430640 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.430617    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"]
Sep 17 21:06:06.551405 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.551370    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:06.551630 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.551452    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:06.551630 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.551478    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:06.551630 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.551528    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcnwh\" (UniqueName: \"kubernetes.io/projected/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kube-api-access-hcnwh\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:06.551805 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.551629    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:06.652787 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.652750    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:06.652951 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.652814    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:06.652951 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.652838    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:06.652951 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.652874    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcnwh\" (UniqueName: \"kubernetes.io/projected/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kube-api-access-hcnwh\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:06.652951 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.652919    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:06.653305 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.653278    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:06.653507 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.653488    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:06.653601 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.653496    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:06.655718 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.655698    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:06.661247 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.661225    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcnwh\" (UniqueName: \"kubernetes.io/projected/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kube-api-access-hcnwh\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:06.728590 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:06.728483    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:07.001992 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:07.001967    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"]
Sep 17 21:06:07.004419 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:06:07.004393    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba6548d_406e_4b1f_b7c5_2372e3a6fe82.slice/crio-227ba1a1e4c7b010be11e451ecdd841d638682bc742c562b3a774918fb83aebf WatchSource:0}: Error finding container 227ba1a1e4c7b010be11e451ecdd841d638682bc742c562b3a774918fb83aebf: Status 404 returned error can't find the container with id 227ba1a1e4c7b010be11e451ecdd841d638682bc742c562b3a774918fb83aebf
Sep 17 21:06:07.006823 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:07.006806    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:06:07.251061 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:07.251030    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" event={"ID":"fba6548d-406e-4b1f-b7c5-2372e3a6fe82","Type":"ContainerStarted","Data":"227ba1a1e4c7b010be11e451ecdd841d638682bc742c562b3a774918fb83aebf"}
Sep 17 21:06:11.273334 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:11.273297    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" event={"ID":"fba6548d-406e-4b1f-b7c5-2372e3a6fe82","Type":"ContainerStarted","Data":"6ace715293935617fca26e314d4130fb9721b781b603d3c2ab338e6e6b8714e1"}
Sep 17 21:06:15.293031 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:15.292992    2571 generic.go:356] "Generic (PLEG): container finished" podID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerID="6ace715293935617fca26e314d4130fb9721b781b603d3c2ab338e6e6b8714e1" exitCode=0
Sep 17 21:06:15.293407 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:15.293062    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" event={"ID":"fba6548d-406e-4b1f-b7c5-2372e3a6fe82","Type":"ContainerDied","Data":"6ace715293935617fca26e314d4130fb9721b781b603d3c2ab338e6e6b8714e1"}
Sep 17 21:06:37.439118 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:37.439081    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" event={"ID":"fba6548d-406e-4b1f-b7c5-2372e3a6fe82","Type":"ContainerStarted","Data":"d600840991909b4d770c6759e6719589c7bdf0133e67e3ae129aecf87d3caa53"}
Sep 17 21:06:40.459461 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:40.459426    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" event={"ID":"fba6548d-406e-4b1f-b7c5-2372e3a6fe82","Type":"ContainerStarted","Data":"69c00e0f54a27da2dcba260d49b1e5df46ece48368f65c3f572527461c1f627c"}
Sep 17 21:06:40.459853 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:40.459497    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:40.459853 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:40.459529    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:06:40.481043 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:40.480994    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" podStartSLOduration=2.608302617 podStartE2EDuration="34.480978705s" totalImagesPullingTime="28.110836264s" totalInitContainerRuntime="3.761839824s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:06:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:06:40.478003864 +0000 UTC m=+1011.982481461" watchObservedRunningTime="2026-09-17 21:06:40.480978705 +0000 UTC m=+1011.985456301"
Sep 17 21:06:46.470862 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:06:46.470831    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:07:06.471961 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:07:06.471912    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 21:07:16.471989 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:07:16.471949    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 21:07:26.472265 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:07:26.472223    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 21:07:36.472370 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:07:36.472282    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 21:07:46.472328 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:07:46.472282    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.41:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 21:07:56.988050 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:07:56.988015    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:08:06.603416 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:06.603381    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"]
Sep 17 21:08:06.603945 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:06.603654    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerName="kserve-container" containerID="cri-o://d600840991909b4d770c6759e6719589c7bdf0133e67e3ae129aecf87d3caa53" gracePeriod=30
Sep 17 21:08:06.603945 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:06.603700    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerName="kube-rbac-proxy" containerID="cri-o://69c00e0f54a27da2dcba260d49b1e5df46ece48368f65c3f572527461c1f627c" gracePeriod=30
Sep 17 21:08:06.887641 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:06.887537    2571 generic.go:356] "Generic (PLEG): container finished" podID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerID="69c00e0f54a27da2dcba260d49b1e5df46ece48368f65c3f572527461c1f627c" exitCode=2
Sep 17 21:08:06.887641 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:06.887592    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" event={"ID":"fba6548d-406e-4b1f-b7c5-2372e3a6fe82","Type":"ContainerDied","Data":"69c00e0f54a27da2dcba260d49b1e5df46ece48368f65c3f572527461c1f627c"}
Sep 17 21:08:11.465576 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.465508    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 17 21:08:11.757412 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.757388    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:08:11.806883 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.806848    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"]
Sep 17 21:08:11.807225 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.807213    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerName="kserve-container"
Sep 17 21:08:11.807273 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.807227    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerName="kserve-container"
Sep 17 21:08:11.807273 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.807239    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerName="kube-rbac-proxy"
Sep 17 21:08:11.807273 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.807244    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerName="kube-rbac-proxy"
Sep 17 21:08:11.807273 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.807251    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerName="storage-initializer"
Sep 17 21:08:11.807273 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.807257    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerName="storage-initializer"
Sep 17 21:08:11.807417 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.807321    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerName="kserve-container"
Sep 17 21:08:11.807417 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.807328    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerName="kube-rbac-proxy"
Sep 17 21:08:11.810676 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.810650    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:11.814126 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.814103    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 17 21:08:11.814786 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.814771    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 21:08:11.845075 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.845045    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"]
Sep 17 21:08:11.866642 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.866615    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcnwh\" (UniqueName: \"kubernetes.io/projected/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kube-api-access-hcnwh\") pod \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") "
Sep 17 21:08:11.866768 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.866669    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kube-rbac-proxy-sar-config\") pod \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") "
Sep 17 21:08:11.866768 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.866704    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-proxy-tls\") pod \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") "
Sep 17 21:08:11.866875 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.866766    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kserve-provision-location\") pod \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") "
Sep 17 21:08:11.866875 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.866842    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-cabundle-cert\") pod \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\" (UID: \"fba6548d-406e-4b1f-b7c5-2372e3a6fe82\") "
Sep 17 21:08:11.866997 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.866973    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15783a77-c2cf-4345-bce1-b1931fbed977-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:11.867057 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.867022    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15783a77-c2cf-4345-bce1-b1931fbed977-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:11.867109 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.867072    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fba6548d-406e-4b1f-b7c5-2372e3a6fe82" (UID: "fba6548d-406e-4b1f-b7c5-2372e3a6fe82"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:08:11.867160 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.867118    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fba6548d-406e-4b1f-b7c5-2372e3a6fe82" (UID: "fba6548d-406e-4b1f-b7c5-2372e3a6fe82"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:08:11.867160 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.867117    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15783a77-c2cf-4345-bce1-b1931fbed977-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:11.867244 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.867173    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fba6548d-406e-4b1f-b7c5-2372e3a6fe82" (UID: "fba6548d-406e-4b1f-b7c5-2372e3a6fe82"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:08:11.867244 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.867197    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5gfm\" (UniqueName: \"kubernetes.io/projected/15783a77-c2cf-4345-bce1-b1931fbed977-kube-api-access-p5gfm\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:11.867315 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.867262    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15783a77-c2cf-4345-bce1-b1931fbed977-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:11.867315 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.867311    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kserve-provision-location\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:08:11.867388 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.867322    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-cabundle-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:08:11.867388 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.867331    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:08:11.869151 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.869126    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fba6548d-406e-4b1f-b7c5-2372e3a6fe82" (UID: "fba6548d-406e-4b1f-b7c5-2372e3a6fe82"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:08:11.869151 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.869136    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kube-api-access-hcnwh" (OuterVolumeSpecName: "kube-api-access-hcnwh") pod "fba6548d-406e-4b1f-b7c5-2372e3a6fe82" (UID: "fba6548d-406e-4b1f-b7c5-2372e3a6fe82"). InnerVolumeSpecName "kube-api-access-hcnwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:08:11.915809 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.915776    2571 generic.go:356] "Generic (PLEG): container finished" podID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" containerID="d600840991909b4d770c6759e6719589c7bdf0133e67e3ae129aecf87d3caa53" exitCode=0
Sep 17 21:08:11.915975 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.915843    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" event={"ID":"fba6548d-406e-4b1f-b7c5-2372e3a6fe82","Type":"ContainerDied","Data":"d600840991909b4d770c6759e6719589c7bdf0133e67e3ae129aecf87d3caa53"}
Sep 17 21:08:11.915975 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.915853    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"
Sep 17 21:08:11.915975 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.915908    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v" event={"ID":"fba6548d-406e-4b1f-b7c5-2372e3a6fe82","Type":"ContainerDied","Data":"227ba1a1e4c7b010be11e451ecdd841d638682bc742c562b3a774918fb83aebf"}
Sep 17 21:08:11.915975 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.915945    2571 scope.go:117] "RemoveContainer" containerID="69c00e0f54a27da2dcba260d49b1e5df46ece48368f65c3f572527461c1f627c"
Sep 17 21:08:11.925369 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.925353    2571 scope.go:117] "RemoveContainer" containerID="d600840991909b4d770c6759e6719589c7bdf0133e67e3ae129aecf87d3caa53"
Sep 17 21:08:11.933484 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.933465    2571 scope.go:117] "RemoveContainer" containerID="6ace715293935617fca26e314d4130fb9721b781b603d3c2ab338e6e6b8714e1"
Sep 17 21:08:11.939415 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.939394    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"]
Sep 17 21:08:11.941413 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.941400    2571 scope.go:117] "RemoveContainer" containerID="69c00e0f54a27da2dcba260d49b1e5df46ece48368f65c3f572527461c1f627c"
Sep 17 21:08:11.941697 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:08:11.941677    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c00e0f54a27da2dcba260d49b1e5df46ece48368f65c3f572527461c1f627c\": container with ID starting with 69c00e0f54a27da2dcba260d49b1e5df46ece48368f65c3f572527461c1f627c not found: ID does not exist" containerID="69c00e0f54a27da2dcba260d49b1e5df46ece48368f65c3f572527461c1f627c"
Sep 17 21:08:11.941763 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.941703    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c00e0f54a27da2dcba260d49b1e5df46ece48368f65c3f572527461c1f627c"} err="failed to get container status \"69c00e0f54a27da2dcba260d49b1e5df46ece48368f65c3f572527461c1f627c\": rpc error: code = NotFound desc = could not find container \"69c00e0f54a27da2dcba260d49b1e5df46ece48368f65c3f572527461c1f627c\": container with ID starting with 69c00e0f54a27da2dcba260d49b1e5df46ece48368f65c3f572527461c1f627c not found: ID does not exist"
Sep 17 21:08:11.941763 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.941720    2571 scope.go:117] "RemoveContainer" containerID="d600840991909b4d770c6759e6719589c7bdf0133e67e3ae129aecf87d3caa53"
Sep 17 21:08:11.941955 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:08:11.941942    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d600840991909b4d770c6759e6719589c7bdf0133e67e3ae129aecf87d3caa53\": container with ID starting with d600840991909b4d770c6759e6719589c7bdf0133e67e3ae129aecf87d3caa53 not found: ID does not exist" containerID="d600840991909b4d770c6759e6719589c7bdf0133e67e3ae129aecf87d3caa53"
Sep 17 21:08:11.941996 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.941960    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d600840991909b4d770c6759e6719589c7bdf0133e67e3ae129aecf87d3caa53"} err="failed to get container status \"d600840991909b4d770c6759e6719589c7bdf0133e67e3ae129aecf87d3caa53\": rpc error: code = NotFound desc = could not find container \"d600840991909b4d770c6759e6719589c7bdf0133e67e3ae129aecf87d3caa53\": container with ID starting with d600840991909b4d770c6759e6719589c7bdf0133e67e3ae129aecf87d3caa53 not found: ID does not exist"
Sep 17 21:08:11.941996 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.941977    2571 scope.go:117] "RemoveContainer" containerID="6ace715293935617fca26e314d4130fb9721b781b603d3c2ab338e6e6b8714e1"
Sep 17 21:08:11.942255 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:08:11.942240    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ace715293935617fca26e314d4130fb9721b781b603d3c2ab338e6e6b8714e1\": container with ID starting with 6ace715293935617fca26e314d4130fb9721b781b603d3c2ab338e6e6b8714e1 not found: ID does not exist" containerID="6ace715293935617fca26e314d4130fb9721b781b603d3c2ab338e6e6b8714e1"
Sep 17 21:08:11.942320 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.942256    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ace715293935617fca26e314d4130fb9721b781b603d3c2ab338e6e6b8714e1"} err="failed to get container status \"6ace715293935617fca26e314d4130fb9721b781b603d3c2ab338e6e6b8714e1\": rpc error: code = NotFound desc = could not find container \"6ace715293935617fca26e314d4130fb9721b781b603d3c2ab338e6e6b8714e1\": container with ID starting with 6ace715293935617fca26e314d4130fb9721b781b603d3c2ab338e6e6b8714e1 not found: ID does not exist"
Sep 17 21:08:11.945539 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.945515    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-mbg4v"]
Sep 17 21:08:11.967941 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.967919    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15783a77-c2cf-4345-bce1-b1931fbed977-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:11.968068 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.967955    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15783a77-c2cf-4345-bce1-b1931fbed977-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:11.968068 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.967978    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15783a77-c2cf-4345-bce1-b1931fbed977-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:11.968068 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.968010    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15783a77-c2cf-4345-bce1-b1931fbed977-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:11.968068 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.968062    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5gfm\" (UniqueName: \"kubernetes.io/projected/15783a77-c2cf-4345-bce1-b1931fbed977-kube-api-access-p5gfm\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:11.968271 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.968106    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcnwh\" (UniqueName: \"kubernetes.io/projected/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-kube-api-access-hcnwh\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:08:11.968271 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.968120    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fba6548d-406e-4b1f-b7c5-2372e3a6fe82-proxy-tls\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:08:11.968515 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.968494    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15783a77-c2cf-4345-bce1-b1931fbed977-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:11.968729 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.968710    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15783a77-c2cf-4345-bce1-b1931fbed977-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:11.968797 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.968741    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15783a77-c2cf-4345-bce1-b1931fbed977-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:11.970492 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.970475    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15783a77-c2cf-4345-bce1-b1931fbed977-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:11.976688 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:11.976666    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5gfm\" (UniqueName: \"kubernetes.io/projected/15783a77-c2cf-4345-bce1-b1931fbed977-kube-api-access-p5gfm\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:12.120595 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:12.120537    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:12.395330 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:12.395252    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"]
Sep 17 21:08:12.398466 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:08:12.398440    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15783a77_c2cf_4345_bce1_b1931fbed977.slice/crio-2b898e3c922bf9c8705c6d979a163561ea5fd233fb1110c255700b9b4ff633c4 WatchSource:0}: Error finding container 2b898e3c922bf9c8705c6d979a163561ea5fd233fb1110c255700b9b4ff633c4: Status 404 returned error can't find the container with id 2b898e3c922bf9c8705c6d979a163561ea5fd233fb1110c255700b9b4ff633c4
Sep 17 21:08:12.670414 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:12.670324    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fba6548d-406e-4b1f-b7c5-2372e3a6fe82" path="/var/lib/kubelet/pods/fba6548d-406e-4b1f-b7c5-2372e3a6fe82/volumes"
Sep 17 21:08:12.922970 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:12.922885    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" event={"ID":"15783a77-c2cf-4345-bce1-b1931fbed977","Type":"ContainerStarted","Data":"b2181c8609b4890fe724797c2756fa3f60fb5826ca0fbf19ff099e98d3b01375"}
Sep 17 21:08:12.922970 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:12.922950    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" event={"ID":"15783a77-c2cf-4345-bce1-b1931fbed977","Type":"ContainerStarted","Data":"2b898e3c922bf9c8705c6d979a163561ea5fd233fb1110c255700b9b4ff633c4"}
Sep 17 21:08:16.943287 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:16.943250    2571 generic.go:356] "Generic (PLEG): container finished" podID="15783a77-c2cf-4345-bce1-b1931fbed977" containerID="b2181c8609b4890fe724797c2756fa3f60fb5826ca0fbf19ff099e98d3b01375" exitCode=0
Sep 17 21:08:16.943823 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:16.943328    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" event={"ID":"15783a77-c2cf-4345-bce1-b1931fbed977","Type":"ContainerDied","Data":"b2181c8609b4890fe724797c2756fa3f60fb5826ca0fbf19ff099e98d3b01375"}
Sep 17 21:08:17.950738 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:17.950708    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" event={"ID":"15783a77-c2cf-4345-bce1-b1931fbed977","Type":"ContainerStarted","Data":"8e3d6fcf6835e60f3867eeef4af7a248b3cdbb7522c63d06813ed013d41a27da"}
Sep 17 21:08:17.951160 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:17.950767    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" event={"ID":"15783a77-c2cf-4345-bce1-b1931fbed977","Type":"ContainerStarted","Data":"f7aed59deb0579f30bdd391de5791f13233804dd4f565280c01a398198c56507"}
Sep 17 21:08:17.951160 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:17.950987    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:17.951160 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:17.951138    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:17.972212 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:17.972155    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" podStartSLOduration=3.005857197 podStartE2EDuration="6.972141413s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.966284216s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:08:17.969816162 +0000 UTC m=+1109.474293760" watchObservedRunningTime="2026-09-17 21:08:17.972141413 +0000 UTC m=+1109.476619008"
Sep 17 21:08:23.962640 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:23.962612    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:08:53.963525 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:08:53.963481    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 21:09:03.963693 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:03.963604    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 21:09:13.963676 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:13.963635    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 21:09:23.963676 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:23.963641    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 21:09:34.486952 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:34.486918    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:09:41.817768 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:41.817725    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"]
Sep 17 21:09:41.818238 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:41.818035    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" containerName="kserve-container" containerID="cri-o://f7aed59deb0579f30bdd391de5791f13233804dd4f565280c01a398198c56507" gracePeriod=30
Sep 17 21:09:41.818238 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:41.818059    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" containerName="kube-rbac-proxy" containerID="cri-o://8e3d6fcf6835e60f3867eeef4af7a248b3cdbb7522c63d06813ed013d41a27da" gracePeriod=30
Sep 17 21:09:42.364477 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:42.364447    2571 generic.go:356] "Generic (PLEG): container finished" podID="15783a77-c2cf-4345-bce1-b1931fbed977" containerID="8e3d6fcf6835e60f3867eeef4af7a248b3cdbb7522c63d06813ed013d41a27da" exitCode=2
Sep 17 21:09:42.364672 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:42.364516    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" event={"ID":"15783a77-c2cf-4345-bce1-b1931fbed977","Type":"ContainerDied","Data":"8e3d6fcf6835e60f3867eeef4af7a248b3cdbb7522c63d06813ed013d41a27da"}
Sep 17 21:09:43.956753 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:43.956708    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 17 21:09:43.963316 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:43.963289    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.42:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 21:09:47.104450 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.104069    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"]
Sep 17 21:09:47.108607 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.108586    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:47.111770 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.111727    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 17 21:09:47.113375 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.113355    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 17 21:09:47.121656 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.121634    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"]
Sep 17 21:09:47.171984 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.171956    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:09:47.222192 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.222159    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5gfm\" (UniqueName: \"kubernetes.io/projected/15783a77-c2cf-4345-bce1-b1931fbed977-kube-api-access-p5gfm\") pod \"15783a77-c2cf-4345-bce1-b1931fbed977\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") "
Sep 17 21:09:47.222192 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.222194    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15783a77-c2cf-4345-bce1-b1931fbed977-proxy-tls\") pod \"15783a77-c2cf-4345-bce1-b1931fbed977\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") "
Sep 17 21:09:47.222395 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.222246    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15783a77-c2cf-4345-bce1-b1931fbed977-cabundle-cert\") pod \"15783a77-c2cf-4345-bce1-b1931fbed977\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") "
Sep 17 21:09:47.222395 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.222303    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15783a77-c2cf-4345-bce1-b1931fbed977-kserve-provision-location\") pod \"15783a77-c2cf-4345-bce1-b1931fbed977\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") "
Sep 17 21:09:47.222486 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.222410    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15783a77-c2cf-4345-bce1-b1931fbed977-kube-rbac-proxy-sar-config\") pod \"15783a77-c2cf-4345-bce1-b1931fbed977\" (UID: \"15783a77-c2cf-4345-bce1-b1931fbed977\") "
Sep 17 21:09:47.222612 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.222590    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37437a32-00a0-4349-8c76-e93c8b0f7e5c-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:47.222710 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.222659    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p8nf\" (UniqueName: \"kubernetes.io/projected/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kube-api-access-9p8nf\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:47.222773 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.222713    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37437a32-00a0-4349-8c76-e93c8b0f7e5c-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:47.222773 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.222753    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:47.222773 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.222748    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15783a77-c2cf-4345-bce1-b1931fbed977-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "15783a77-c2cf-4345-bce1-b1931fbed977" (UID: "15783a77-c2cf-4345-bce1-b1931fbed977"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:09:47.222930 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.222783    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15783a77-c2cf-4345-bce1-b1931fbed977-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "15783a77-c2cf-4345-bce1-b1931fbed977" (UID: "15783a77-c2cf-4345-bce1-b1931fbed977"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:09:47.222930 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.222780    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:47.222930 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.222843    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15783a77-c2cf-4345-bce1-b1931fbed977-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15783a77-c2cf-4345-bce1-b1931fbed977" (UID: "15783a77-c2cf-4345-bce1-b1931fbed977"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:09:47.222930 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.222922    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15783a77-c2cf-4345-bce1-b1931fbed977-cabundle-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:09:47.223071 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.222941    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15783a77-c2cf-4345-bce1-b1931fbed977-kserve-provision-location\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:09:47.224614 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.224587    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15783a77-c2cf-4345-bce1-b1931fbed977-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15783a77-c2cf-4345-bce1-b1931fbed977" (UID: "15783a77-c2cf-4345-bce1-b1931fbed977"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:09:47.224714 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.224634    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15783a77-c2cf-4345-bce1-b1931fbed977-kube-api-access-p5gfm" (OuterVolumeSpecName: "kube-api-access-p5gfm") pod "15783a77-c2cf-4345-bce1-b1931fbed977" (UID: "15783a77-c2cf-4345-bce1-b1931fbed977"). InnerVolumeSpecName "kube-api-access-p5gfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:09:47.323453 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.323421    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37437a32-00a0-4349-8c76-e93c8b0f7e5c-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:47.323646 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.323473    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p8nf\" (UniqueName: \"kubernetes.io/projected/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kube-api-access-9p8nf\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:47.323646 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.323505    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37437a32-00a0-4349-8c76-e93c8b0f7e5c-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:47.323646 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.323537    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:47.323646 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.323593    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:47.323811 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.323671    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15783a77-c2cf-4345-bce1-b1931fbed977-proxy-tls\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:09:47.323811 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.323689    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15783a77-c2cf-4345-bce1-b1931fbed977-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:09:47.323811 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.323704    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5gfm\" (UniqueName: \"kubernetes.io/projected/15783a77-c2cf-4345-bce1-b1931fbed977-kube-api-access-p5gfm\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:09:47.323811 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:09:47.323685    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 17 21:09:47.323811 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:09:47.323785    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37437a32-00a0-4349-8c76-e93c8b0f7e5c-proxy-tls podName:37437a32-00a0-4349-8c76-e93c8b0f7e5c nodeName:}" failed. No retries permitted until 2026-09-17 21:09:47.823763101 +0000 UTC m=+1199.328240689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/37437a32-00a0-4349-8c76-e93c8b0f7e5c-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" (UID: "37437a32-00a0-4349-8c76-e93c8b0f7e5c") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 17 21:09:47.324074 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.323994    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:47.324207 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.324188    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:47.324243 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.324193    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37437a32-00a0-4349-8c76-e93c8b0f7e5c-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:47.335399 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.335374    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p8nf\" (UniqueName: \"kubernetes.io/projected/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kube-api-access-9p8nf\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:47.389535 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.389497    2571 generic.go:356] "Generic (PLEG): container finished" podID="15783a77-c2cf-4345-bce1-b1931fbed977" containerID="f7aed59deb0579f30bdd391de5791f13233804dd4f565280c01a398198c56507" exitCode=0
Sep 17 21:09:47.389706 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.389579    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"
Sep 17 21:09:47.389706 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.389584    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" event={"ID":"15783a77-c2cf-4345-bce1-b1931fbed977","Type":"ContainerDied","Data":"f7aed59deb0579f30bdd391de5791f13233804dd4f565280c01a398198c56507"}
Sep 17 21:09:47.389706 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.389651    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b" event={"ID":"15783a77-c2cf-4345-bce1-b1931fbed977","Type":"ContainerDied","Data":"2b898e3c922bf9c8705c6d979a163561ea5fd233fb1110c255700b9b4ff633c4"}
Sep 17 21:09:47.389706 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.389678    2571 scope.go:117] "RemoveContainer" containerID="8e3d6fcf6835e60f3867eeef4af7a248b3cdbb7522c63d06813ed013d41a27da"
Sep 17 21:09:47.399128 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.399109    2571 scope.go:117] "RemoveContainer" containerID="f7aed59deb0579f30bdd391de5791f13233804dd4f565280c01a398198c56507"
Sep 17 21:09:47.407067 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.407050    2571 scope.go:117] "RemoveContainer" containerID="b2181c8609b4890fe724797c2756fa3f60fb5826ca0fbf19ff099e98d3b01375"
Sep 17 21:09:47.413450 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.413424    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"]
Sep 17 21:09:47.415708 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.415686    2571 scope.go:117] "RemoveContainer" containerID="8e3d6fcf6835e60f3867eeef4af7a248b3cdbb7522c63d06813ed013d41a27da"
Sep 17 21:09:47.416079 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:09:47.416054    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3d6fcf6835e60f3867eeef4af7a248b3cdbb7522c63d06813ed013d41a27da\": container with ID starting with 8e3d6fcf6835e60f3867eeef4af7a248b3cdbb7522c63d06813ed013d41a27da not found: ID does not exist" containerID="8e3d6fcf6835e60f3867eeef4af7a248b3cdbb7522c63d06813ed013d41a27da"
Sep 17 21:09:47.416160 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.416080    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3d6fcf6835e60f3867eeef4af7a248b3cdbb7522c63d06813ed013d41a27da"} err="failed to get container status \"8e3d6fcf6835e60f3867eeef4af7a248b3cdbb7522c63d06813ed013d41a27da\": rpc error: code = NotFound desc = could not find container \"8e3d6fcf6835e60f3867eeef4af7a248b3cdbb7522c63d06813ed013d41a27da\": container with ID starting with 8e3d6fcf6835e60f3867eeef4af7a248b3cdbb7522c63d06813ed013d41a27da not found: ID does not exist"
Sep 17 21:09:47.416160 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.416097    2571 scope.go:117] "RemoveContainer" containerID="f7aed59deb0579f30bdd391de5791f13233804dd4f565280c01a398198c56507"
Sep 17 21:09:47.416381 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:09:47.416361    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7aed59deb0579f30bdd391de5791f13233804dd4f565280c01a398198c56507\": container with ID starting with f7aed59deb0579f30bdd391de5791f13233804dd4f565280c01a398198c56507 not found: ID does not exist" containerID="f7aed59deb0579f30bdd391de5791f13233804dd4f565280c01a398198c56507"
Sep 17 21:09:47.416434 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.416397    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7aed59deb0579f30bdd391de5791f13233804dd4f565280c01a398198c56507"} err="failed to get container status \"f7aed59deb0579f30bdd391de5791f13233804dd4f565280c01a398198c56507\": rpc error: code = NotFound desc = could not find container \"f7aed59deb0579f30bdd391de5791f13233804dd4f565280c01a398198c56507\": container with ID starting with f7aed59deb0579f30bdd391de5791f13233804dd4f565280c01a398198c56507 not found: ID does not exist"
Sep 17 21:09:47.416434 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.416412    2571 scope.go:117] "RemoveContainer" containerID="b2181c8609b4890fe724797c2756fa3f60fb5826ca0fbf19ff099e98d3b01375"
Sep 17 21:09:47.416724 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:09:47.416704    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2181c8609b4890fe724797c2756fa3f60fb5826ca0fbf19ff099e98d3b01375\": container with ID starting with b2181c8609b4890fe724797c2756fa3f60fb5826ca0fbf19ff099e98d3b01375 not found: ID does not exist" containerID="b2181c8609b4890fe724797c2756fa3f60fb5826ca0fbf19ff099e98d3b01375"
Sep 17 21:09:47.416793 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.416728    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2181c8609b4890fe724797c2756fa3f60fb5826ca0fbf19ff099e98d3b01375"} err="failed to get container status \"b2181c8609b4890fe724797c2756fa3f60fb5826ca0fbf19ff099e98d3b01375\": rpc error: code = NotFound desc = could not find container \"b2181c8609b4890fe724797c2756fa3f60fb5826ca0fbf19ff099e98d3b01375\": container with ID starting with b2181c8609b4890fe724797c2756fa3f60fb5826ca0fbf19ff099e98d3b01375 not found: ID does not exist"
Sep 17 21:09:47.417233 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.417213    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jb74b"]
Sep 17 21:09:47.827127 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.827089    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37437a32-00a0-4349-8c76-e93c8b0f7e5c-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:47.829648 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:47.829628    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37437a32-00a0-4349-8c76-e93c8b0f7e5c-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:48.020200 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:48.020159    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:48.295623 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:48.295594    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"]
Sep 17 21:09:48.297470 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:09:48.297448    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37437a32_00a0_4349_8c76_e93c8b0f7e5c.slice/crio-3c7ca419d2168b9325daef743787a6a6790ecb8af7a40b18a459fa9c7f29399f WatchSource:0}: Error finding container 3c7ca419d2168b9325daef743787a6a6790ecb8af7a40b18a459fa9c7f29399f: Status 404 returned error can't find the container with id 3c7ca419d2168b9325daef743787a6a6790ecb8af7a40b18a459fa9c7f29399f
Sep 17 21:09:48.396997 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:48.396961    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" event={"ID":"37437a32-00a0-4349-8c76-e93c8b0f7e5c","Type":"ContainerStarted","Data":"81ca86e36a0ca65fd217a48a3b1077b266bc330a22360baa898c29535d8097d1"}
Sep 17 21:09:48.397164 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:48.397033    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" event={"ID":"37437a32-00a0-4349-8c76-e93c8b0f7e5c","Type":"ContainerStarted","Data":"3c7ca419d2168b9325daef743787a6a6790ecb8af7a40b18a459fa9c7f29399f"}
Sep 17 21:09:48.677940 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:48.672221    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" path="/var/lib/kubelet/pods/15783a77-c2cf-4345-bce1-b1931fbed977/volumes"
Sep 17 21:09:52.421039 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:52.421009    2571 generic.go:356] "Generic (PLEG): container finished" podID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerID="81ca86e36a0ca65fd217a48a3b1077b266bc330a22360baa898c29535d8097d1" exitCode=0
Sep 17 21:09:52.421534 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:52.421089    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" event={"ID":"37437a32-00a0-4349-8c76-e93c8b0f7e5c","Type":"ContainerDied","Data":"81ca86e36a0ca65fd217a48a3b1077b266bc330a22360baa898c29535d8097d1"}
Sep 17 21:09:53.427736 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:53.427698    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" event={"ID":"37437a32-00a0-4349-8c76-e93c8b0f7e5c","Type":"ContainerStarted","Data":"a8d689bb5e8f76262daaac1a74aa083f50490a1b82e880704a1f97c7b95f9b56"}
Sep 17 21:09:53.428122 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:53.427789    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" event={"ID":"37437a32-00a0-4349-8c76-e93c8b0f7e5c","Type":"ContainerStarted","Data":"09ddecf5b2522980af0e678f848fac16e93a2ec02c2bcd64cd1650c0a063d655"}
Sep 17 21:09:53.428122 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:53.428011    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:53.428234 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:53.428180    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:09:53.452751 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:53.452707    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" podStartSLOduration=2.634448016 podStartE2EDuration="6.452693355s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.818245339s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:09:53.450918326 +0000 UTC m=+1204.955395924" watchObservedRunningTime="2026-09-17 21:09:53.452693355 +0000 UTC m=+1204.957170950"
Sep 17 21:09:59.438834 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:09:59.438802    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:10:29.440151 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:10:29.440108    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 21:10:39.439567 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:10:39.439517    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 21:10:49.439867 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:10:49.439827    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 21:10:59.440280 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:10:59.440231    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 21:10:59.664451 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:10:59.664411    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 21:11:09.665303 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:09.665259    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 21:11:20.186562 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:20.186526    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:11:27.122059 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:27.122028    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"]
Sep 17 21:11:27.122588 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:27.122327    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="kserve-container" containerID="cri-o://09ddecf5b2522980af0e678f848fac16e93a2ec02c2bcd64cd1650c0a063d655" gracePeriod=30
Sep 17 21:11:27.122588 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:27.122381    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="kube-rbac-proxy" containerID="cri-o://a8d689bb5e8f76262daaac1a74aa083f50490a1b82e880704a1f97c7b95f9b56" gracePeriod=30
Sep 17 21:11:27.888924 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:27.888891    2571 generic.go:356] "Generic (PLEG): container finished" podID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerID="a8d689bb5e8f76262daaac1a74aa083f50490a1b82e880704a1f97c7b95f9b56" exitCode=2
Sep 17 21:11:27.889100 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:27.888962    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" event={"ID":"37437a32-00a0-4349-8c76-e93c8b0f7e5c","Type":"ContainerDied","Data":"a8d689bb5e8f76262daaac1a74aa083f50490a1b82e880704a1f97c7b95f9b56"}
Sep 17 21:11:29.434270 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:29.434229    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 21:11:29.664978 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:29.664937    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.43:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 21:11:33.574329 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.574294    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:11:33.627147 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.627103    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kserve-provision-location\") pod \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") "
Sep 17 21:11:33.627147 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.627150    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kube-rbac-proxy-sar-config\") pod \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") "
Sep 17 21:11:33.627403 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.627183    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37437a32-00a0-4349-8c76-e93c8b0f7e5c-proxy-tls\") pod \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") "
Sep 17 21:11:33.627403 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.627244    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37437a32-00a0-4349-8c76-e93c8b0f7e5c-cabundle-cert\") pod \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") "
Sep 17 21:11:33.627403 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.627268    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9p8nf\" (UniqueName: \"kubernetes.io/projected/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kube-api-access-9p8nf\") pod \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\" (UID: \"37437a32-00a0-4349-8c76-e93c8b0f7e5c\") "
Sep 17 21:11:33.627594 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.627537    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37437a32-00a0-4349-8c76-e93c8b0f7e5c" (UID: "37437a32-00a0-4349-8c76-e93c8b0f7e5c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:11:33.627657 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.627595    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "37437a32-00a0-4349-8c76-e93c8b0f7e5c" (UID: "37437a32-00a0-4349-8c76-e93c8b0f7e5c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:11:33.627701 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.627678    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37437a32-00a0-4349-8c76-e93c8b0f7e5c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "37437a32-00a0-4349-8c76-e93c8b0f7e5c" (UID: "37437a32-00a0-4349-8c76-e93c8b0f7e5c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:11:33.629596 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.629565    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kube-api-access-9p8nf" (OuterVolumeSpecName: "kube-api-access-9p8nf") pod "37437a32-00a0-4349-8c76-e93c8b0f7e5c" (UID: "37437a32-00a0-4349-8c76-e93c8b0f7e5c"). InnerVolumeSpecName "kube-api-access-9p8nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:11:33.629596 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.629590    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37437a32-00a0-4349-8c76-e93c8b0f7e5c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "37437a32-00a0-4349-8c76-e93c8b0f7e5c" (UID: "37437a32-00a0-4349-8c76-e93c8b0f7e5c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:11:33.729030 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.728926    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37437a32-00a0-4349-8c76-e93c8b0f7e5c-cabundle-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:11:33.729030 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.728963    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p8nf\" (UniqueName: \"kubernetes.io/projected/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kube-api-access-9p8nf\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:11:33.729030 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.728976    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kserve-provision-location\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:11:33.729030 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.728988    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37437a32-00a0-4349-8c76-e93c8b0f7e5c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:11:33.729030 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.728999    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37437a32-00a0-4349-8c76-e93c8b0f7e5c-proxy-tls\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:11:33.924753 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.924719    2571 generic.go:356] "Generic (PLEG): container finished" podID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerID="09ddecf5b2522980af0e678f848fac16e93a2ec02c2bcd64cd1650c0a063d655" exitCode=0
Sep 17 21:11:33.924753 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.924759    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" event={"ID":"37437a32-00a0-4349-8c76-e93c8b0f7e5c","Type":"ContainerDied","Data":"09ddecf5b2522980af0e678f848fac16e93a2ec02c2bcd64cd1650c0a063d655"}
Sep 17 21:11:33.924957 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.924779    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"
Sep 17 21:11:33.924957 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.924807    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6" event={"ID":"37437a32-00a0-4349-8c76-e93c8b0f7e5c","Type":"ContainerDied","Data":"3c7ca419d2168b9325daef743787a6a6790ecb8af7a40b18a459fa9c7f29399f"}
Sep 17 21:11:33.924957 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.924841    2571 scope.go:117] "RemoveContainer" containerID="a8d689bb5e8f76262daaac1a74aa083f50490a1b82e880704a1f97c7b95f9b56"
Sep 17 21:11:33.934125 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.934108    2571 scope.go:117] "RemoveContainer" containerID="09ddecf5b2522980af0e678f848fac16e93a2ec02c2bcd64cd1650c0a063d655"
Sep 17 21:11:33.941906 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.941887    2571 scope.go:117] "RemoveContainer" containerID="81ca86e36a0ca65fd217a48a3b1077b266bc330a22360baa898c29535d8097d1"
Sep 17 21:11:33.948117 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.948092    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"]
Sep 17 21:11:33.950846 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.950829    2571 scope.go:117] "RemoveContainer" containerID="a8d689bb5e8f76262daaac1a74aa083f50490a1b82e880704a1f97c7b95f9b56"
Sep 17 21:11:33.951126 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:11:33.951107    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d689bb5e8f76262daaac1a74aa083f50490a1b82e880704a1f97c7b95f9b56\": container with ID starting with a8d689bb5e8f76262daaac1a74aa083f50490a1b82e880704a1f97c7b95f9b56 not found: ID does not exist" containerID="a8d689bb5e8f76262daaac1a74aa083f50490a1b82e880704a1f97c7b95f9b56"
Sep 17 21:11:33.951194 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.951135    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d689bb5e8f76262daaac1a74aa083f50490a1b82e880704a1f97c7b95f9b56"} err="failed to get container status \"a8d689bb5e8f76262daaac1a74aa083f50490a1b82e880704a1f97c7b95f9b56\": rpc error: code = NotFound desc = could not find container \"a8d689bb5e8f76262daaac1a74aa083f50490a1b82e880704a1f97c7b95f9b56\": container with ID starting with a8d689bb5e8f76262daaac1a74aa083f50490a1b82e880704a1f97c7b95f9b56 not found: ID does not exist"
Sep 17 21:11:33.951194 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.951158    2571 scope.go:117] "RemoveContainer" containerID="09ddecf5b2522980af0e678f848fac16e93a2ec02c2bcd64cd1650c0a063d655"
Sep 17 21:11:33.951460 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:11:33.951442    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ddecf5b2522980af0e678f848fac16e93a2ec02c2bcd64cd1650c0a063d655\": container with ID starting with 09ddecf5b2522980af0e678f848fac16e93a2ec02c2bcd64cd1650c0a063d655 not found: ID does not exist" containerID="09ddecf5b2522980af0e678f848fac16e93a2ec02c2bcd64cd1650c0a063d655"
Sep 17 21:11:33.951519 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.951465    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ddecf5b2522980af0e678f848fac16e93a2ec02c2bcd64cd1650c0a063d655"} err="failed to get container status \"09ddecf5b2522980af0e678f848fac16e93a2ec02c2bcd64cd1650c0a063d655\": rpc error: code = NotFound desc = could not find container \"09ddecf5b2522980af0e678f848fac16e93a2ec02c2bcd64cd1650c0a063d655\": container with ID starting with 09ddecf5b2522980af0e678f848fac16e93a2ec02c2bcd64cd1650c0a063d655 not found: ID does not exist"
Sep 17 21:11:33.951519 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.951480    2571 scope.go:117] "RemoveContainer" containerID="81ca86e36a0ca65fd217a48a3b1077b266bc330a22360baa898c29535d8097d1"
Sep 17 21:11:33.951782 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:11:33.951764    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ca86e36a0ca65fd217a48a3b1077b266bc330a22360baa898c29535d8097d1\": container with ID starting with 81ca86e36a0ca65fd217a48a3b1077b266bc330a22360baa898c29535d8097d1 not found: ID does not exist" containerID="81ca86e36a0ca65fd217a48a3b1077b266bc330a22360baa898c29535d8097d1"
Sep 17 21:11:33.951845 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.951789    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ca86e36a0ca65fd217a48a3b1077b266bc330a22360baa898c29535d8097d1"} err="failed to get container status \"81ca86e36a0ca65fd217a48a3b1077b266bc330a22360baa898c29535d8097d1\": rpc error: code = NotFound desc = could not find container \"81ca86e36a0ca65fd217a48a3b1077b266bc330a22360baa898c29535d8097d1\": container with ID starting with 81ca86e36a0ca65fd217a48a3b1077b266bc330a22360baa898c29535d8097d1 not found: ID does not exist"
Sep 17 21:11:33.952291 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:33.952270    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-msbk6"]
Sep 17 21:11:34.675743 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:11:34.675707    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" path="/var/lib/kubelet/pods/37437a32-00a0-4349-8c76-e93c8b0f7e5c/volumes"
Sep 17 21:13:05.152768 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.152733    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"]
Sep 17 21:13:05.153242 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.153094    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" containerName="kserve-container"
Sep 17 21:13:05.153242 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.153105    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" containerName="kserve-container"
Sep 17 21:13:05.153242 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.153118    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" containerName="kube-rbac-proxy"
Sep 17 21:13:05.153242 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.153124    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" containerName="kube-rbac-proxy"
Sep 17 21:13:05.153242 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.153142    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="storage-initializer"
Sep 17 21:13:05.153242 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.153148    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="storage-initializer"
Sep 17 21:13:05.153242 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.153157    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" containerName="storage-initializer"
Sep 17 21:13:05.153242 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.153162    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" containerName="storage-initializer"
Sep 17 21:13:05.153242 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.153170    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="kserve-container"
Sep 17 21:13:05.153242 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.153175    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="kserve-container"
Sep 17 21:13:05.153242 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.153183    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="kube-rbac-proxy"
Sep 17 21:13:05.153242 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.153192    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="kube-rbac-proxy"
Sep 17 21:13:05.153648 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.153251    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" containerName="kserve-container"
Sep 17 21:13:05.153648 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.153259    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="kserve-container"
Sep 17 21:13:05.153648 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.153266    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="37437a32-00a0-4349-8c76-e93c8b0f7e5c" containerName="kube-rbac-proxy"
Sep 17 21:13:05.153648 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.153271    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="15783a77-c2cf-4345-bce1-b1931fbed977" containerName="kube-rbac-proxy"
Sep 17 21:13:05.156722 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.156702    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:05.159590 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.159571    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 17 21:13:05.159708 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.159680    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:13:05.160504 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.160487    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:13:05.160589 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.160498    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:13:05.160589 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.160507    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:13:05.160589 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.160579    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:13:05.160708 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.160596    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 17 21:13:05.170699 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.170678    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"]
Sep 17 21:13:05.226861 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.226827    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a647bbbb-fb9d-4171-9250-18dce4d9945d-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:05.226861 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.226863    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a647bbbb-fb9d-4171-9250-18dce4d9945d-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:05.227052 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.226892    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a647bbbb-fb9d-4171-9250-18dce4d9945d-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:05.227052 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.226942    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a647bbbb-fb9d-4171-9250-18dce4d9945d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:05.227052 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.226987    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lfnp\" (UniqueName: \"kubernetes.io/projected/a647bbbb-fb9d-4171-9250-18dce4d9945d-kube-api-access-7lfnp\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:05.328165 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.328112    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a647bbbb-fb9d-4171-9250-18dce4d9945d-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:05.328165 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.328173    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a647bbbb-fb9d-4171-9250-18dce4d9945d-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:05.328420 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.328208    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a647bbbb-fb9d-4171-9250-18dce4d9945d-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:05.328420 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.328227    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a647bbbb-fb9d-4171-9250-18dce4d9945d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:05.328420 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.328250    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lfnp\" (UniqueName: \"kubernetes.io/projected/a647bbbb-fb9d-4171-9250-18dce4d9945d-kube-api-access-7lfnp\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:05.328420 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:13:05.328280    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 17 21:13:05.328420 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:13:05.328359    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a647bbbb-fb9d-4171-9250-18dce4d9945d-proxy-tls podName:a647bbbb-fb9d-4171-9250-18dce4d9945d nodeName:}" failed. No retries permitted until 2026-09-17 21:13:05.828337535 +0000 UTC m=+1397.332815122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a647bbbb-fb9d-4171-9250-18dce4d9945d-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v" (UID: "a647bbbb-fb9d-4171-9250-18dce4d9945d") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 17 21:13:05.328731 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.328686    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a647bbbb-fb9d-4171-9250-18dce4d9945d-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:05.328947 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.328925    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a647bbbb-fb9d-4171-9250-18dce4d9945d-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:05.329057 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.329033    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a647bbbb-fb9d-4171-9250-18dce4d9945d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:05.337328 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.337306    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lfnp\" (UniqueName: \"kubernetes.io/projected/a647bbbb-fb9d-4171-9250-18dce4d9945d-kube-api-access-7lfnp\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:05.833588 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.833524    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a647bbbb-fb9d-4171-9250-18dce4d9945d-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:05.836081 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:05.836060    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a647bbbb-fb9d-4171-9250-18dce4d9945d-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:06.068105 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:06.068064    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:13:06.343804 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:06.343777    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"]
Sep 17 21:13:06.346335 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:13:06.346304    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda647bbbb_fb9d_4171_9250_18dce4d9945d.slice/crio-6e7783ecfb61cd472b7e169466d7b382c2a61d075338eb9a754482de622a36f5 WatchSource:0}: Error finding container 6e7783ecfb61cd472b7e169466d7b382c2a61d075338eb9a754482de622a36f5: Status 404 returned error can't find the container with id 6e7783ecfb61cd472b7e169466d7b382c2a61d075338eb9a754482de622a36f5
Sep 17 21:13:06.381215 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:06.381181    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v" event={"ID":"a647bbbb-fb9d-4171-9250-18dce4d9945d","Type":"ContainerStarted","Data":"6e7783ecfb61cd472b7e169466d7b382c2a61d075338eb9a754482de622a36f5"}
Sep 17 21:13:07.387864 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:07.387820    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v" event={"ID":"a647bbbb-fb9d-4171-9250-18dce4d9945d","Type":"ContainerStarted","Data":"d739e69920eefd90c3924e8860ed81c322a542f71e7ee9d8433d237eae119f6c"}
Sep 17 21:13:36.528811 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:36.528775    2571 generic.go:356] "Generic (PLEG): container finished" podID="a647bbbb-fb9d-4171-9250-18dce4d9945d" containerID="d739e69920eefd90c3924e8860ed81c322a542f71e7ee9d8433d237eae119f6c" exitCode=0
Sep 17 21:13:36.529214 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:36.528848    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v" event={"ID":"a647bbbb-fb9d-4171-9250-18dce4d9945d","Type":"ContainerDied","Data":"d739e69920eefd90c3924e8860ed81c322a542f71e7ee9d8433d237eae119f6c"}
Sep 17 21:13:36.530246 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:13:36.530228    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:15:36.216158 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:15:36.216118    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v" event={"ID":"a647bbbb-fb9d-4171-9250-18dce4d9945d","Type":"ContainerStarted","Data":"fd02f104bc3901ad1eaf2c34e6a5f5b4df9bfc72daead96d6d1e96919d184ffd"}
Sep 17 21:15:36.216639 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:15:36.216194    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v" event={"ID":"a647bbbb-fb9d-4171-9250-18dce4d9945d","Type":"ContainerStarted","Data":"fe3f76495090ce2c394f89e5498f2f2b7a62657a29050601dd8a308b8e452b55"}
Sep 17 21:15:36.216639 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:15:36.216230    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:15:36.216639 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:15:36.216260    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:15:36.238793 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:15:36.238729    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v" podStartSLOduration=2.602757376 podStartE2EDuration="2m31.238712265s" totalImagesPullingTime="1m58.684502418s" totalInitContainerRuntime="29.951452471s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:15:36.236460856 +0000 UTC m=+1547.740938467" watchObservedRunningTime="2026-09-17 21:15:36.238712265 +0000 UTC m=+1547.743189860"
Sep 17 21:15:42.228133 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:15:42.228100    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:17:12.700860 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:12.700823    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:17:21.661968 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:21.661931    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"]
Sep 17 21:17:21.662446 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:21.662274    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v" podUID="a647bbbb-fb9d-4171-9250-18dce4d9945d" containerName="kserve-container" containerID="cri-o://fe3f76495090ce2c394f89e5498f2f2b7a62657a29050601dd8a308b8e452b55" gracePeriod=30
Sep 17 21:17:21.662446 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:21.662370    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v" podUID="a647bbbb-fb9d-4171-9250-18dce4d9945d" containerName="kube-rbac-proxy" containerID="cri-o://fd02f104bc3901ad1eaf2c34e6a5f5b4df9bfc72daead96d6d1e96919d184ffd" gracePeriod=30
Sep 17 21:17:22.222624 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:22.222579    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v" podUID="a647bbbb-fb9d-4171-9250-18dce4d9945d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 17 21:17:22.664797 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:22.664752    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v" podUID="a647bbbb-fb9d-4171-9250-18dce4d9945d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.44:8080/v2/models/isvc-autogluon-v2-noslash/ready\": dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 21:17:22.751014 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:22.750968    2571 generic.go:356] "Generic (PLEG): container finished" podID="a647bbbb-fb9d-4171-9250-18dce4d9945d" containerID="fd02f104bc3901ad1eaf2c34e6a5f5b4df9bfc72daead96d6d1e96919d184ffd" exitCode=2
Sep 17 21:17:22.751220 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:22.751037    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v" event={"ID":"a647bbbb-fb9d-4171-9250-18dce4d9945d","Type":"ContainerDied","Data":"fd02f104bc3901ad1eaf2c34e6a5f5b4df9bfc72daead96d6d1e96919d184ffd"}
Sep 17 21:17:23.116886 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.116862    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:17:23.220702 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.220608    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a647bbbb-fb9d-4171-9250-18dce4d9945d-proxy-tls\") pod \"a647bbbb-fb9d-4171-9250-18dce4d9945d\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") "
Sep 17 21:17:23.220702 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.220648    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lfnp\" (UniqueName: \"kubernetes.io/projected/a647bbbb-fb9d-4171-9250-18dce4d9945d-kube-api-access-7lfnp\") pod \"a647bbbb-fb9d-4171-9250-18dce4d9945d\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") "
Sep 17 21:17:23.220702 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.220675    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a647bbbb-fb9d-4171-9250-18dce4d9945d-cabundle-cert\") pod \"a647bbbb-fb9d-4171-9250-18dce4d9945d\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") "
Sep 17 21:17:23.220948 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.220749    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a647bbbb-fb9d-4171-9250-18dce4d9945d-kserve-provision-location\") pod \"a647bbbb-fb9d-4171-9250-18dce4d9945d\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") "
Sep 17 21:17:23.220948 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.220769    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a647bbbb-fb9d-4171-9250-18dce4d9945d-kube-rbac-proxy-sar-config\") pod \"a647bbbb-fb9d-4171-9250-18dce4d9945d\" (UID: \"a647bbbb-fb9d-4171-9250-18dce4d9945d\") "
Sep 17 21:17:23.221119 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.221083    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a647bbbb-fb9d-4171-9250-18dce4d9945d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a647bbbb-fb9d-4171-9250-18dce4d9945d" (UID: "a647bbbb-fb9d-4171-9250-18dce4d9945d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:17:23.221244 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.221207    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a647bbbb-fb9d-4171-9250-18dce4d9945d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a647bbbb-fb9d-4171-9250-18dce4d9945d" (UID: "a647bbbb-fb9d-4171-9250-18dce4d9945d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:17:23.223100 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.223065    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a647bbbb-fb9d-4171-9250-18dce4d9945d-kube-api-access-7lfnp" (OuterVolumeSpecName: "kube-api-access-7lfnp") pod "a647bbbb-fb9d-4171-9250-18dce4d9945d" (UID: "a647bbbb-fb9d-4171-9250-18dce4d9945d"). InnerVolumeSpecName "kube-api-access-7lfnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:17:23.223205 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.223107    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a647bbbb-fb9d-4171-9250-18dce4d9945d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a647bbbb-fb9d-4171-9250-18dce4d9945d" (UID: "a647bbbb-fb9d-4171-9250-18dce4d9945d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:17:23.225361 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.225337    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a647bbbb-fb9d-4171-9250-18dce4d9945d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a647bbbb-fb9d-4171-9250-18dce4d9945d" (UID: "a647bbbb-fb9d-4171-9250-18dce4d9945d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:17:23.321652 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.321612    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a647bbbb-fb9d-4171-9250-18dce4d9945d-kserve-provision-location\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:17:23.321652 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.321649    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a647bbbb-fb9d-4171-9250-18dce4d9945d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:17:23.321652 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.321662    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a647bbbb-fb9d-4171-9250-18dce4d9945d-proxy-tls\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:17:23.321903 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.321678    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lfnp\" (UniqueName: \"kubernetes.io/projected/a647bbbb-fb9d-4171-9250-18dce4d9945d-kube-api-access-7lfnp\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:17:23.321903 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.321689    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a647bbbb-fb9d-4171-9250-18dce4d9945d-cabundle-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:17:23.758144 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.758110    2571 generic.go:356] "Generic (PLEG): container finished" podID="a647bbbb-fb9d-4171-9250-18dce4d9945d" containerID="fe3f76495090ce2c394f89e5498f2f2b7a62657a29050601dd8a308b8e452b55" exitCode=0
Sep 17 21:17:23.758658 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.758174    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"
Sep 17 21:17:23.758658 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.758186    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v" event={"ID":"a647bbbb-fb9d-4171-9250-18dce4d9945d","Type":"ContainerDied","Data":"fe3f76495090ce2c394f89e5498f2f2b7a62657a29050601dd8a308b8e452b55"}
Sep 17 21:17:23.758658 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.758233    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v" event={"ID":"a647bbbb-fb9d-4171-9250-18dce4d9945d","Type":"ContainerDied","Data":"6e7783ecfb61cd472b7e169466d7b382c2a61d075338eb9a754482de622a36f5"}
Sep 17 21:17:23.758658 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.758268    2571 scope.go:117] "RemoveContainer" containerID="fd02f104bc3901ad1eaf2c34e6a5f5b4df9bfc72daead96d6d1e96919d184ffd"
Sep 17 21:17:23.767590 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.767573    2571 scope.go:117] "RemoveContainer" containerID="fe3f76495090ce2c394f89e5498f2f2b7a62657a29050601dd8a308b8e452b55"
Sep 17 21:17:23.775674 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.775652    2571 scope.go:117] "RemoveContainer" containerID="d739e69920eefd90c3924e8860ed81c322a542f71e7ee9d8433d237eae119f6c"
Sep 17 21:17:23.782828 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.782803    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"]
Sep 17 21:17:23.783834 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.783819    2571 scope.go:117] "RemoveContainer" containerID="fd02f104bc3901ad1eaf2c34e6a5f5b4df9bfc72daead96d6d1e96919d184ffd"
Sep 17 21:17:23.784079 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:17:23.784061    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd02f104bc3901ad1eaf2c34e6a5f5b4df9bfc72daead96d6d1e96919d184ffd\": container with ID starting with fd02f104bc3901ad1eaf2c34e6a5f5b4df9bfc72daead96d6d1e96919d184ffd not found: ID does not exist" containerID="fd02f104bc3901ad1eaf2c34e6a5f5b4df9bfc72daead96d6d1e96919d184ffd"
Sep 17 21:17:23.784140 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.784086    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd02f104bc3901ad1eaf2c34e6a5f5b4df9bfc72daead96d6d1e96919d184ffd"} err="failed to get container status \"fd02f104bc3901ad1eaf2c34e6a5f5b4df9bfc72daead96d6d1e96919d184ffd\": rpc error: code = NotFound desc = could not find container \"fd02f104bc3901ad1eaf2c34e6a5f5b4df9bfc72daead96d6d1e96919d184ffd\": container with ID starting with fd02f104bc3901ad1eaf2c34e6a5f5b4df9bfc72daead96d6d1e96919d184ffd not found: ID does not exist"
Sep 17 21:17:23.784140 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.784104    2571 scope.go:117] "RemoveContainer" containerID="fe3f76495090ce2c394f89e5498f2f2b7a62657a29050601dd8a308b8e452b55"
Sep 17 21:17:23.784330 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:17:23.784315    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3f76495090ce2c394f89e5498f2f2b7a62657a29050601dd8a308b8e452b55\": container with ID starting with fe3f76495090ce2c394f89e5498f2f2b7a62657a29050601dd8a308b8e452b55 not found: ID does not exist" containerID="fe3f76495090ce2c394f89e5498f2f2b7a62657a29050601dd8a308b8e452b55"
Sep 17 21:17:23.784375 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.784335    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3f76495090ce2c394f89e5498f2f2b7a62657a29050601dd8a308b8e452b55"} err="failed to get container status \"fe3f76495090ce2c394f89e5498f2f2b7a62657a29050601dd8a308b8e452b55\": rpc error: code = NotFound desc = could not find container \"fe3f76495090ce2c394f89e5498f2f2b7a62657a29050601dd8a308b8e452b55\": container with ID starting with fe3f76495090ce2c394f89e5498f2f2b7a62657a29050601dd8a308b8e452b55 not found: ID does not exist"
Sep 17 21:17:23.784375 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.784348    2571 scope.go:117] "RemoveContainer" containerID="d739e69920eefd90c3924e8860ed81c322a542f71e7ee9d8433d237eae119f6c"
Sep 17 21:17:23.784639 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:17:23.784621    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d739e69920eefd90c3924e8860ed81c322a542f71e7ee9d8433d237eae119f6c\": container with ID starting with d739e69920eefd90c3924e8860ed81c322a542f71e7ee9d8433d237eae119f6c not found: ID does not exist" containerID="d739e69920eefd90c3924e8860ed81c322a542f71e7ee9d8433d237eae119f6c"
Sep 17 21:17:23.784688 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.784648    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d739e69920eefd90c3924e8860ed81c322a542f71e7ee9d8433d237eae119f6c"} err="failed to get container status \"d739e69920eefd90c3924e8860ed81c322a542f71e7ee9d8433d237eae119f6c\": rpc error: code = NotFound desc = could not find container \"d739e69920eefd90c3924e8860ed81c322a542f71e7ee9d8433d237eae119f6c\": container with ID starting with d739e69920eefd90c3924e8860ed81c322a542f71e7ee9d8433d237eae119f6c not found: ID does not exist"
Sep 17 21:17:23.790842 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:23.790821    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9l75v"]
Sep 17 21:17:24.671161 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:24.671127    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a647bbbb-fb9d-4171-9250-18dce4d9945d" path="/var/lib/kubelet/pods/a647bbbb-fb9d-4171-9250-18dce4d9945d/volumes"
Sep 17 21:17:26.848494 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.848458    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"]
Sep 17 21:17:26.849121 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.849097    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a647bbbb-fb9d-4171-9250-18dce4d9945d" containerName="kube-rbac-proxy"
Sep 17 21:17:26.849207 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.849124    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a647bbbb-fb9d-4171-9250-18dce4d9945d" containerName="kube-rbac-proxy"
Sep 17 21:17:26.849207 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.849138    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a647bbbb-fb9d-4171-9250-18dce4d9945d" containerName="storage-initializer"
Sep 17 21:17:26.849207 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.849147    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a647bbbb-fb9d-4171-9250-18dce4d9945d" containerName="storage-initializer"
Sep 17 21:17:26.849207 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.849172    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a647bbbb-fb9d-4171-9250-18dce4d9945d" containerName="kserve-container"
Sep 17 21:17:26.849207 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.849180    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a647bbbb-fb9d-4171-9250-18dce4d9945d" containerName="kserve-container"
Sep 17 21:17:26.849466 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.849268    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a647bbbb-fb9d-4171-9250-18dce4d9945d" containerName="kube-rbac-proxy"
Sep 17 21:17:26.849466 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.849279    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a647bbbb-fb9d-4171-9250-18dce4d9945d" containerName="kserve-container"
Sep 17 21:17:26.852724 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.852704    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:26.855429 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.855407    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 17 21:17:26.855655 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.855632    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:17:26.855769 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.855723    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:17:26.855769 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.855763    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 17 21:17:26.856577 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.856534    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:17:26.856674 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.856578    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:17:26.856741 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.856677    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:17:26.862596 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.862575    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"]
Sep 17 21:17:26.954983 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.954945    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:26.954983 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.954983    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:26.955197 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.955082    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bwtg\" (UniqueName: \"kubernetes.io/projected/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kube-api-access-7bwtg\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:26.955197 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.955144    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:26.955197 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:26.955189    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:27.056412 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:27.056357    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:27.056412 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:27.056426    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:27.056680 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:27.056446    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:27.056680 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:27.056489    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bwtg\" (UniqueName: \"kubernetes.io/projected/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kube-api-access-7bwtg\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:27.056680 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:27.056527    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:27.056680 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:17:27.056641    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 17 21:17:27.056876 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:17:27.056733    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-proxy-tls podName:5fe76d89-96ef-4c33-a116-cd7fbe85e9bc nodeName:}" failed. No retries permitted until 2026-09-17 21:17:27.556708096 +0000 UTC m=+1659.061185690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc" (UID: "5fe76d89-96ef-4c33-a116-cd7fbe85e9bc") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 17 21:17:27.056945 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:27.056869    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:27.057139 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:27.057122    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:27.057185 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:27.057149    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:27.065961 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:27.065939    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bwtg\" (UniqueName: \"kubernetes.io/projected/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kube-api-access-7bwtg\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:27.560309 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:27.560273    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:27.562923 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:27.562897    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:27.764294 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:27.764258    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:28.039310 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:28.039280    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"]
Sep 17 21:17:28.041311 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:17:28.041281    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe76d89_96ef_4c33_a116_cd7fbe85e9bc.slice/crio-d2c67649ef0c77cff48f6be0002689b84d1cf802bb73cb763a56afae6571cd0e WatchSource:0}: Error finding container d2c67649ef0c77cff48f6be0002689b84d1cf802bb73cb763a56afae6571cd0e: Status 404 returned error can't find the container with id d2c67649ef0c77cff48f6be0002689b84d1cf802bb73cb763a56afae6571cd0e
Sep 17 21:17:28.786014 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:28.785970    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc" event={"ID":"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc","Type":"ContainerStarted","Data":"d43e3ddcc0a91139a319a25cdf616e7bfe2a6b6de0390714da75773d6c2af976"}
Sep 17 21:17:28.786208 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:28.786045    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc" event={"ID":"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc","Type":"ContainerStarted","Data":"d2c67649ef0c77cff48f6be0002689b84d1cf802bb73cb763a56afae6571cd0e"}
Sep 17 21:17:37.833116 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:37.833081    2571 generic.go:356] "Generic (PLEG): container finished" podID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" containerID="d43e3ddcc0a91139a319a25cdf616e7bfe2a6b6de0390714da75773d6c2af976" exitCode=0
Sep 17 21:17:37.833570 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:37.833164    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc" event={"ID":"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc","Type":"ContainerDied","Data":"d43e3ddcc0a91139a319a25cdf616e7bfe2a6b6de0390714da75773d6c2af976"}
Sep 17 21:17:38.841014 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:38.840971    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc" event={"ID":"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc","Type":"ContainerStarted","Data":"2ead4fa237c867a995b6e6973fd03034ecc5567425544e94d03ce549b83af603"}
Sep 17 21:17:38.841377 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:38.841088    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc" event={"ID":"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc","Type":"ContainerStarted","Data":"d85904d87ae614ce1f1e08ab9898f7a05d1e3315ad5325b1929fd353d05e39eb"}
Sep 17 21:17:38.841377 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:38.841308    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:38.864300 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:38.864240    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc" podStartSLOduration=4.025554979 podStartE2EDuration="12.864220308s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.838665329s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:17:38.862418569 +0000 UTC m=+1670.366896178" watchObservedRunningTime="2026-09-17 21:17:38.864220308 +0000 UTC m=+1670.368697905"
Sep 17 21:17:39.846613 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:39.846576    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:39.848325 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:39.848296    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc" podUID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 21:17:40.851672 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:40.851632    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc" podUID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 21:17:45.856783 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:45.856756    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:17:45.857275 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:45.857233    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc" podUID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 21:17:55.858754 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:17:55.858717    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:18:13.632936 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:13.632848    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"]
Sep 17 21:18:13.633481 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:13.633112    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc" podUID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" containerName="kserve-container" containerID="cri-o://d85904d87ae614ce1f1e08ab9898f7a05d1e3315ad5325b1929fd353d05e39eb" gracePeriod=30
Sep 17 21:18:13.633481 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:13.633245    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc" podUID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" containerName="kube-rbac-proxy" containerID="cri-o://2ead4fa237c867a995b6e6973fd03034ecc5567425544e94d03ce549b83af603" gracePeriod=30
Sep 17 21:18:14.025195 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:14.025101    2571 generic.go:356] "Generic (PLEG): container finished" podID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" containerID="2ead4fa237c867a995b6e6973fd03034ecc5567425544e94d03ce549b83af603" exitCode=2
Sep 17 21:18:14.025195 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:14.025153    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc" event={"ID":"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc","Type":"ContainerDied","Data":"2ead4fa237c867a995b6e6973fd03034ecc5567425544e94d03ce549b83af603"}
Sep 17 21:18:15.481292 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:15.481264    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:18:15.586229 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:15.586191    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bwtg\" (UniqueName: \"kubernetes.io/projected/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kube-api-access-7bwtg\") pod \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") "
Sep 17 21:18:15.586411 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:15.586255    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kserve-provision-location\") pod \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") "
Sep 17 21:18:15.586411 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:15.586295    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kube-rbac-proxy-sar-config\") pod \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") "
Sep 17 21:18:15.586411 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:15.586363    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-cabundle-cert\") pod \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") "
Sep 17 21:18:15.586638 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:15.586437    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-proxy-tls\") pod \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\" (UID: \"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc\") "
Sep 17 21:18:15.586788 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:15.586759    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" (UID: "5fe76d89-96ef-4c33-a116-cd7fbe85e9bc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:18:15.586849 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:15.586786    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" (UID: "5fe76d89-96ef-4c33-a116-cd7fbe85e9bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:18:15.587287 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:15.587265    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" (UID: "5fe76d89-96ef-4c33-a116-cd7fbe85e9bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:18:15.588665 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:15.588641    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" (UID: "5fe76d89-96ef-4c33-a116-cd7fbe85e9bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:18:15.588754 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:15.588705    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kube-api-access-7bwtg" (OuterVolumeSpecName: "kube-api-access-7bwtg") pod "5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" (UID: "5fe76d89-96ef-4c33-a116-cd7fbe85e9bc"). InnerVolumeSpecName "kube-api-access-7bwtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:18:15.687251 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:15.687216    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-cabundle-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:18:15.687251 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:15.687246    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-proxy-tls\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:18:15.687251 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:15.687255    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bwtg\" (UniqueName: \"kubernetes.io/projected/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kube-api-access-7bwtg\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:18:15.687488 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:15.687266    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kserve-provision-location\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:18:15.687488 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:15.687277    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:18:16.038228 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:16.038143    2571 generic.go:356] "Generic (PLEG): container finished" podID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" containerID="d85904d87ae614ce1f1e08ab9898f7a05d1e3315ad5325b1929fd353d05e39eb" exitCode=0
Sep 17 21:18:16.038228 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:16.038212    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"
Sep 17 21:18:16.038485 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:16.038222    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc" event={"ID":"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc","Type":"ContainerDied","Data":"d85904d87ae614ce1f1e08ab9898f7a05d1e3315ad5325b1929fd353d05e39eb"}
Sep 17 21:18:16.038485 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:16.038324    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc" event={"ID":"5fe76d89-96ef-4c33-a116-cd7fbe85e9bc","Type":"ContainerDied","Data":"d2c67649ef0c77cff48f6be0002689b84d1cf802bb73cb763a56afae6571cd0e"}
Sep 17 21:18:16.038485 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:16.038353    2571 scope.go:117] "RemoveContainer" containerID="2ead4fa237c867a995b6e6973fd03034ecc5567425544e94d03ce549b83af603"
Sep 17 21:18:16.047543 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:16.047525    2571 scope.go:117] "RemoveContainer" containerID="d85904d87ae614ce1f1e08ab9898f7a05d1e3315ad5325b1929fd353d05e39eb"
Sep 17 21:18:16.055409 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:16.055389    2571 scope.go:117] "RemoveContainer" containerID="d43e3ddcc0a91139a319a25cdf616e7bfe2a6b6de0390714da75773d6c2af976"
Sep 17 21:18:16.061879 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:16.061855    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"]
Sep 17 21:18:16.068233 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:16.068212    2571 scope.go:117] "RemoveContainer" containerID="2ead4fa237c867a995b6e6973fd03034ecc5567425544e94d03ce549b83af603"
Sep 17 21:18:16.068334 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:16.068284    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-5m7zc"]
Sep 17 21:18:16.068617 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:18:16.068597    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ead4fa237c867a995b6e6973fd03034ecc5567425544e94d03ce549b83af603\": container with ID starting with 2ead4fa237c867a995b6e6973fd03034ecc5567425544e94d03ce549b83af603 not found: ID does not exist" containerID="2ead4fa237c867a995b6e6973fd03034ecc5567425544e94d03ce549b83af603"
Sep 17 21:18:16.068668 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:16.068626    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ead4fa237c867a995b6e6973fd03034ecc5567425544e94d03ce549b83af603"} err="failed to get container status \"2ead4fa237c867a995b6e6973fd03034ecc5567425544e94d03ce549b83af603\": rpc error: code = NotFound desc = could not find container \"2ead4fa237c867a995b6e6973fd03034ecc5567425544e94d03ce549b83af603\": container with ID starting with 2ead4fa237c867a995b6e6973fd03034ecc5567425544e94d03ce549b83af603 not found: ID does not exist"
Sep 17 21:18:16.068668 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:16.068645    2571 scope.go:117] "RemoveContainer" containerID="d85904d87ae614ce1f1e08ab9898f7a05d1e3315ad5325b1929fd353d05e39eb"
Sep 17 21:18:16.068904 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:18:16.068890    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d85904d87ae614ce1f1e08ab9898f7a05d1e3315ad5325b1929fd353d05e39eb\": container with ID starting with d85904d87ae614ce1f1e08ab9898f7a05d1e3315ad5325b1929fd353d05e39eb not found: ID does not exist" containerID="d85904d87ae614ce1f1e08ab9898f7a05d1e3315ad5325b1929fd353d05e39eb"
Sep 17 21:18:16.068954 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:16.068908    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d85904d87ae614ce1f1e08ab9898f7a05d1e3315ad5325b1929fd353d05e39eb"} err="failed to get container status \"d85904d87ae614ce1f1e08ab9898f7a05d1e3315ad5325b1929fd353d05e39eb\": rpc error: code = NotFound desc = could not find container \"d85904d87ae614ce1f1e08ab9898f7a05d1e3315ad5325b1929fd353d05e39eb\": container with ID starting with d85904d87ae614ce1f1e08ab9898f7a05d1e3315ad5325b1929fd353d05e39eb not found: ID does not exist"
Sep 17 21:18:16.068954 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:16.068922    2571 scope.go:117] "RemoveContainer" containerID="d43e3ddcc0a91139a319a25cdf616e7bfe2a6b6de0390714da75773d6c2af976"
Sep 17 21:18:16.069187 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:18:16.069168    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d43e3ddcc0a91139a319a25cdf616e7bfe2a6b6de0390714da75773d6c2af976\": container with ID starting with d43e3ddcc0a91139a319a25cdf616e7bfe2a6b6de0390714da75773d6c2af976 not found: ID does not exist" containerID="d43e3ddcc0a91139a319a25cdf616e7bfe2a6b6de0390714da75773d6c2af976"
Sep 17 21:18:16.069231 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:16.069196    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43e3ddcc0a91139a319a25cdf616e7bfe2a6b6de0390714da75773d6c2af976"} err="failed to get container status \"d43e3ddcc0a91139a319a25cdf616e7bfe2a6b6de0390714da75773d6c2af976\": rpc error: code = NotFound desc = could not find container \"d43e3ddcc0a91139a319a25cdf616e7bfe2a6b6de0390714da75773d6c2af976\": container with ID starting with d43e3ddcc0a91139a319a25cdf616e7bfe2a6b6de0390714da75773d6c2af976 not found: ID does not exist"
Sep 17 21:18:16.669643 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:16.669610    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" path="/var/lib/kubelet/pods/5fe76d89-96ef-4c33-a116-cd7fbe85e9bc/volumes"
Sep 17 21:18:18.864569 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.864527    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"]
Sep 17 21:18:18.865081 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.865062    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" containerName="kube-rbac-proxy"
Sep 17 21:18:18.865164 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.865085    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" containerName="kube-rbac-proxy"
Sep 17 21:18:18.865164 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.865101    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" containerName="storage-initializer"
Sep 17 21:18:18.865164 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.865111    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" containerName="storage-initializer"
Sep 17 21:18:18.865164 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.865155    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" containerName="kserve-container"
Sep 17 21:18:18.865164 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.865164    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" containerName="kserve-container"
Sep 17 21:18:18.865417 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.865254    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" containerName="kube-rbac-proxy"
Sep 17 21:18:18.865417 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.865273    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe76d89-96ef-4c33-a116-cd7fbe85e9bc" containerName="kserve-container"
Sep 17 21:18:18.868498 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.868475    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:18.872860 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.872830    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:18:18.872860 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.872831    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:18:18.873157 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.873140    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:18:18.873243 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.873204    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 17 21:18:18.873310 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.873298    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 17 21:18:18.874073 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.874024    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:18:18.874198 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.874100    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:18:18.886866 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.886842    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"]
Sep 17 21:18:18.923136 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.923102    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be66cf47-b48f-4897-8a0b-68b85529279b-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:18.923285 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.923146    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be66cf47-b48f-4897-8a0b-68b85529279b-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:18.923285 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.923237    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be66cf47-b48f-4897-8a0b-68b85529279b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:18.923372 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.923280    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl89r\" (UniqueName: \"kubernetes.io/projected/be66cf47-b48f-4897-8a0b-68b85529279b-kube-api-access-sl89r\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:18.923372 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:18.923316    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be66cf47-b48f-4897-8a0b-68b85529279b-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:19.023837 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:19.023804    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be66cf47-b48f-4897-8a0b-68b85529279b-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:19.023837 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:19.023840    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be66cf47-b48f-4897-8a0b-68b85529279b-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:19.024067 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:19.023959    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be66cf47-b48f-4897-8a0b-68b85529279b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:19.024067 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:19.023995    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl89r\" (UniqueName: \"kubernetes.io/projected/be66cf47-b48f-4897-8a0b-68b85529279b-kube-api-access-sl89r\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:19.024067 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:19.024030    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be66cf47-b48f-4897-8a0b-68b85529279b-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:19.024252 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:19.024161    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be66cf47-b48f-4897-8a0b-68b85529279b-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:19.024516 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:19.024492    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be66cf47-b48f-4897-8a0b-68b85529279b-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:19.024700 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:19.024679    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be66cf47-b48f-4897-8a0b-68b85529279b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:19.026527 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:19.026501    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be66cf47-b48f-4897-8a0b-68b85529279b-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:19.036652 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:19.036625    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl89r\" (UniqueName: \"kubernetes.io/projected/be66cf47-b48f-4897-8a0b-68b85529279b-kube-api-access-sl89r\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:19.179674 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:19.179579    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:19.465348 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:19.465319    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"]
Sep 17 21:18:19.466688 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:18:19.466663    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe66cf47_b48f_4897_8a0b_68b85529279b.slice/crio-c4c09e60439cba8f0a4ba0fe7e58292b1c9fa8f215fb66b30e300a65428fde7f WatchSource:0}: Error finding container c4c09e60439cba8f0a4ba0fe7e58292b1c9fa8f215fb66b30e300a65428fde7f: Status 404 returned error can't find the container with id c4c09e60439cba8f0a4ba0fe7e58292b1c9fa8f215fb66b30e300a65428fde7f
Sep 17 21:18:20.058908 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:20.058868    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs" event={"ID":"be66cf47-b48f-4897-8a0b-68b85529279b","Type":"ContainerStarted","Data":"1cae66ee95d5d234e7ce7c1f8f2b98929f9122677dcd93bf38eaddea5801fce5"}
Sep 17 21:18:20.059381 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:20.058933    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs" event={"ID":"be66cf47-b48f-4897-8a0b-68b85529279b","Type":"ContainerStarted","Data":"c4c09e60439cba8f0a4ba0fe7e58292b1c9fa8f215fb66b30e300a65428fde7f"}
Sep 17 21:18:25.083659 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:25.083604    2571 generic.go:356] "Generic (PLEG): container finished" podID="be66cf47-b48f-4897-8a0b-68b85529279b" containerID="1cae66ee95d5d234e7ce7c1f8f2b98929f9122677dcd93bf38eaddea5801fce5" exitCode=0
Sep 17 21:18:25.084043 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:25.083683    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs" event={"ID":"be66cf47-b48f-4897-8a0b-68b85529279b","Type":"ContainerDied","Data":"1cae66ee95d5d234e7ce7c1f8f2b98929f9122677dcd93bf38eaddea5801fce5"}
Sep 17 21:18:26.091272 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:26.091240    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs" event={"ID":"be66cf47-b48f-4897-8a0b-68b85529279b","Type":"ContainerStarted","Data":"85674906cc36c5565b14170ae84cd0910f82de728164dc289503043fab833422"}
Sep 17 21:18:26.091766 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:26.091314    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs" event={"ID":"be66cf47-b48f-4897-8a0b-68b85529279b","Type":"ContainerStarted","Data":"88dcf7f2a47dcb9432a4c9f6900051550f5f82c407a47791d7f5c8bc25aeb40c"}
Sep 17 21:18:26.091766 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:26.091528    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:26.112614 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:26.112535    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs" podStartSLOduration=3.486946784 podStartE2EDuration="8.112519866s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.625573082s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:18:26.111105239 +0000 UTC m=+1717.615582861" watchObservedRunningTime="2026-09-17 21:18:26.112519866 +0000 UTC m=+1717.616997463"
Sep 17 21:18:27.095927 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:27.095898    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:27.097513 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:27.097474    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs" podUID="be66cf47-b48f-4897-8a0b-68b85529279b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 21:18:28.100645 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:28.100602    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs" podUID="be66cf47-b48f-4897-8a0b-68b85529279b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 21:18:33.105435 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:33.105405    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:33.106011 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:33.105977    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs" podUID="be66cf47-b48f-4897-8a0b-68b85529279b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 21:18:43.106726 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:43.106693    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:54.394646 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:54.394609    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"]
Sep 17 21:18:54.395151 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:54.394857    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs" podUID="be66cf47-b48f-4897-8a0b-68b85529279b" containerName="kserve-container" containerID="cri-o://88dcf7f2a47dcb9432a4c9f6900051550f5f82c407a47791d7f5c8bc25aeb40c" gracePeriod=30
Sep 17 21:18:54.395151 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:54.394899    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs" podUID="be66cf47-b48f-4897-8a0b-68b85529279b" containerName="kube-rbac-proxy" containerID="cri-o://85674906cc36c5565b14170ae84cd0910f82de728164dc289503043fab833422" gracePeriod=30
Sep 17 21:18:55.238541 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:55.238500    2571 generic.go:356] "Generic (PLEG): container finished" podID="be66cf47-b48f-4897-8a0b-68b85529279b" containerID="85674906cc36c5565b14170ae84cd0910f82de728164dc289503043fab833422" exitCode=2
Sep 17 21:18:55.238733 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:55.238578    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs" event={"ID":"be66cf47-b48f-4897-8a0b-68b85529279b","Type":"ContainerDied","Data":"85674906cc36c5565b14170ae84cd0910f82de728164dc289503043fab833422"}
Sep 17 21:18:55.843478 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:55.843453    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:55.939228 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:55.939149    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be66cf47-b48f-4897-8a0b-68b85529279b-proxy-tls\") pod \"be66cf47-b48f-4897-8a0b-68b85529279b\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") "
Sep 17 21:18:55.939228 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:55.939216    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be66cf47-b48f-4897-8a0b-68b85529279b-kserve-provision-location\") pod \"be66cf47-b48f-4897-8a0b-68b85529279b\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") "
Sep 17 21:18:55.939415 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:55.939236    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be66cf47-b48f-4897-8a0b-68b85529279b-cabundle-cert\") pod \"be66cf47-b48f-4897-8a0b-68b85529279b\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") "
Sep 17 21:18:55.939415 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:55.939258    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl89r\" (UniqueName: \"kubernetes.io/projected/be66cf47-b48f-4897-8a0b-68b85529279b-kube-api-access-sl89r\") pod \"be66cf47-b48f-4897-8a0b-68b85529279b\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") "
Sep 17 21:18:55.939415 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:55.939331    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be66cf47-b48f-4897-8a0b-68b85529279b-kube-rbac-proxy-sar-config\") pod \"be66cf47-b48f-4897-8a0b-68b85529279b\" (UID: \"be66cf47-b48f-4897-8a0b-68b85529279b\") "
Sep 17 21:18:55.939634 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:55.939599    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be66cf47-b48f-4897-8a0b-68b85529279b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be66cf47-b48f-4897-8a0b-68b85529279b" (UID: "be66cf47-b48f-4897-8a0b-68b85529279b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:18:55.939842 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:55.939819    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be66cf47-b48f-4897-8a0b-68b85529279b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "be66cf47-b48f-4897-8a0b-68b85529279b" (UID: "be66cf47-b48f-4897-8a0b-68b85529279b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:18:55.940210 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:55.940191    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be66cf47-b48f-4897-8a0b-68b85529279b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be66cf47-b48f-4897-8a0b-68b85529279b" (UID: "be66cf47-b48f-4897-8a0b-68b85529279b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:18:55.941431 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:55.941411    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be66cf47-b48f-4897-8a0b-68b85529279b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be66cf47-b48f-4897-8a0b-68b85529279b" (UID: "be66cf47-b48f-4897-8a0b-68b85529279b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:18:55.941524 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:55.941503    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be66cf47-b48f-4897-8a0b-68b85529279b-kube-api-access-sl89r" (OuterVolumeSpecName: "kube-api-access-sl89r") pod "be66cf47-b48f-4897-8a0b-68b85529279b" (UID: "be66cf47-b48f-4897-8a0b-68b85529279b"). InnerVolumeSpecName "kube-api-access-sl89r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:18:56.040388 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.040354    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be66cf47-b48f-4897-8a0b-68b85529279b-kserve-provision-location\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:18:56.040388 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.040382    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be66cf47-b48f-4897-8a0b-68b85529279b-cabundle-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:18:56.040388 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.040392    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl89r\" (UniqueName: \"kubernetes.io/projected/be66cf47-b48f-4897-8a0b-68b85529279b-kube-api-access-sl89r\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:18:56.040735 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.040401    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be66cf47-b48f-4897-8a0b-68b85529279b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:18:56.040735 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.040412    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be66cf47-b48f-4897-8a0b-68b85529279b-proxy-tls\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:18:56.245640 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.245533    2571 generic.go:356] "Generic (PLEG): container finished" podID="be66cf47-b48f-4897-8a0b-68b85529279b" containerID="88dcf7f2a47dcb9432a4c9f6900051550f5f82c407a47791d7f5c8bc25aeb40c" exitCode=0
Sep 17 21:18:56.245640 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.245584    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs" event={"ID":"be66cf47-b48f-4897-8a0b-68b85529279b","Type":"ContainerDied","Data":"88dcf7f2a47dcb9432a4c9f6900051550f5f82c407a47791d7f5c8bc25aeb40c"}
Sep 17 21:18:56.245640 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.245600    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"
Sep 17 21:18:56.245881 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.245645    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs" event={"ID":"be66cf47-b48f-4897-8a0b-68b85529279b","Type":"ContainerDied","Data":"c4c09e60439cba8f0a4ba0fe7e58292b1c9fa8f215fb66b30e300a65428fde7f"}
Sep 17 21:18:56.245881 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.245675    2571 scope.go:117] "RemoveContainer" containerID="85674906cc36c5565b14170ae84cd0910f82de728164dc289503043fab833422"
Sep 17 21:18:56.255375 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.255356    2571 scope.go:117] "RemoveContainer" containerID="88dcf7f2a47dcb9432a4c9f6900051550f5f82c407a47791d7f5c8bc25aeb40c"
Sep 17 21:18:56.266869 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.266610    2571 scope.go:117] "RemoveContainer" containerID="1cae66ee95d5d234e7ce7c1f8f2b98929f9122677dcd93bf38eaddea5801fce5"
Sep 17 21:18:56.268334 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.268312    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"]
Sep 17 21:18:56.274222 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.274197    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z7hzs"]
Sep 17 21:18:56.275564 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.275536    2571 scope.go:117] "RemoveContainer" containerID="85674906cc36c5565b14170ae84cd0910f82de728164dc289503043fab833422"
Sep 17 21:18:56.275835 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:18:56.275819    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85674906cc36c5565b14170ae84cd0910f82de728164dc289503043fab833422\": container with ID starting with 85674906cc36c5565b14170ae84cd0910f82de728164dc289503043fab833422 not found: ID does not exist" containerID="85674906cc36c5565b14170ae84cd0910f82de728164dc289503043fab833422"
Sep 17 21:18:56.275899 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.275845    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85674906cc36c5565b14170ae84cd0910f82de728164dc289503043fab833422"} err="failed to get container status \"85674906cc36c5565b14170ae84cd0910f82de728164dc289503043fab833422\": rpc error: code = NotFound desc = could not find container \"85674906cc36c5565b14170ae84cd0910f82de728164dc289503043fab833422\": container with ID starting with 85674906cc36c5565b14170ae84cd0910f82de728164dc289503043fab833422 not found: ID does not exist"
Sep 17 21:18:56.275899 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.275862    2571 scope.go:117] "RemoveContainer" containerID="88dcf7f2a47dcb9432a4c9f6900051550f5f82c407a47791d7f5c8bc25aeb40c"
Sep 17 21:18:56.276096 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:18:56.276078    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88dcf7f2a47dcb9432a4c9f6900051550f5f82c407a47791d7f5c8bc25aeb40c\": container with ID starting with 88dcf7f2a47dcb9432a4c9f6900051550f5f82c407a47791d7f5c8bc25aeb40c not found: ID does not exist" containerID="88dcf7f2a47dcb9432a4c9f6900051550f5f82c407a47791d7f5c8bc25aeb40c"
Sep 17 21:18:56.276142 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.276110    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88dcf7f2a47dcb9432a4c9f6900051550f5f82c407a47791d7f5c8bc25aeb40c"} err="failed to get container status \"88dcf7f2a47dcb9432a4c9f6900051550f5f82c407a47791d7f5c8bc25aeb40c\": rpc error: code = NotFound desc = could not find container \"88dcf7f2a47dcb9432a4c9f6900051550f5f82c407a47791d7f5c8bc25aeb40c\": container with ID starting with 88dcf7f2a47dcb9432a4c9f6900051550f5f82c407a47791d7f5c8bc25aeb40c not found: ID does not exist"
Sep 17 21:18:56.276142 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.276126    2571 scope.go:117] "RemoveContainer" containerID="1cae66ee95d5d234e7ce7c1f8f2b98929f9122677dcd93bf38eaddea5801fce5"
Sep 17 21:18:56.276404 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:18:56.276387    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cae66ee95d5d234e7ce7c1f8f2b98929f9122677dcd93bf38eaddea5801fce5\": container with ID starting with 1cae66ee95d5d234e7ce7c1f8f2b98929f9122677dcd93bf38eaddea5801fce5 not found: ID does not exist" containerID="1cae66ee95d5d234e7ce7c1f8f2b98929f9122677dcd93bf38eaddea5801fce5"
Sep 17 21:18:56.276467 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.276414    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cae66ee95d5d234e7ce7c1f8f2b98929f9122677dcd93bf38eaddea5801fce5"} err="failed to get container status \"1cae66ee95d5d234e7ce7c1f8f2b98929f9122677dcd93bf38eaddea5801fce5\": rpc error: code = NotFound desc = could not find container \"1cae66ee95d5d234e7ce7c1f8f2b98929f9122677dcd93bf38eaddea5801fce5\": container with ID starting with 1cae66ee95d5d234e7ce7c1f8f2b98929f9122677dcd93bf38eaddea5801fce5 not found: ID does not exist"
Sep 17 21:18:56.676329 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:56.676295    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be66cf47-b48f-4897-8a0b-68b85529279b" path="/var/lib/kubelet/pods/be66cf47-b48f-4897-8a0b-68b85529279b/volumes"
Sep 17 21:18:59.610982 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.610939    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"]
Sep 17 21:18:59.611508 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.611485    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be66cf47-b48f-4897-8a0b-68b85529279b" containerName="storage-initializer"
Sep 17 21:18:59.611641 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.611512    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="be66cf47-b48f-4897-8a0b-68b85529279b" containerName="storage-initializer"
Sep 17 21:18:59.611641 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.611529    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be66cf47-b48f-4897-8a0b-68b85529279b" containerName="kserve-container"
Sep 17 21:18:59.611641 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.611537    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="be66cf47-b48f-4897-8a0b-68b85529279b" containerName="kserve-container"
Sep 17 21:18:59.611641 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.611582    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be66cf47-b48f-4897-8a0b-68b85529279b" containerName="kube-rbac-proxy"
Sep 17 21:18:59.611641 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.611592    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="be66cf47-b48f-4897-8a0b-68b85529279b" containerName="kube-rbac-proxy"
Sep 17 21:18:59.611914 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.611686    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="be66cf47-b48f-4897-8a0b-68b85529279b" containerName="kube-rbac-proxy"
Sep 17 21:18:59.611914 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.611699    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="be66cf47-b48f-4897-8a0b-68b85529279b" containerName="kserve-container"
Sep 17 21:18:59.616973 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.616949    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:18:59.621466 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.621441    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 21:18:59.622274 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.622238    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:18:59.622274 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.622243    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:18:59.622448 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.622309    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:18:59.622448 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.622243    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 21:18:59.623647 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.623622    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:18:59.623747 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.623679    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:18:59.631531 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.631506    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"]
Sep 17 21:18:59.672019 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.671982    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e070d212-336a-49ac-9325-ba9d59839be6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-97jvx\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:18:59.672221 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.672034    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e070d212-336a-49ac-9325-ba9d59839be6-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-97jvx\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:18:59.672221 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.672135    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e070d212-336a-49ac-9325-ba9d59839be6-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-97jvx\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:18:59.672221 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.672182    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkdmc\" (UniqueName: \"kubernetes.io/projected/e070d212-336a-49ac-9325-ba9d59839be6-kube-api-access-kkdmc\") pod \"isvc-canary-create-predictor-7c7f45c769-97jvx\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:18:59.672221 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.672206    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e070d212-336a-49ac-9325-ba9d59839be6-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-97jvx\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:18:59.773229 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.773194    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e070d212-336a-49ac-9325-ba9d59839be6-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-97jvx\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:18:59.773410 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.773239    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkdmc\" (UniqueName: \"kubernetes.io/projected/e070d212-336a-49ac-9325-ba9d59839be6-kube-api-access-kkdmc\") pod \"isvc-canary-create-predictor-7c7f45c769-97jvx\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:18:59.773410 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.773261    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e070d212-336a-49ac-9325-ba9d59839be6-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-97jvx\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:18:59.773410 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.773313    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e070d212-336a-49ac-9325-ba9d59839be6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-97jvx\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:18:59.773410 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.773377    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e070d212-336a-49ac-9325-ba9d59839be6-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-97jvx\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:18:59.773820 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.773788    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e070d212-336a-49ac-9325-ba9d59839be6-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-97jvx\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:18:59.774102 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.774082    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e070d212-336a-49ac-9325-ba9d59839be6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-97jvx\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:18:59.774181 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.774119    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e070d212-336a-49ac-9325-ba9d59839be6-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-97jvx\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:18:59.776085 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.776067    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e070d212-336a-49ac-9325-ba9d59839be6-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-97jvx\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:18:59.784015 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.783995    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkdmc\" (UniqueName: \"kubernetes.io/projected/e070d212-336a-49ac-9325-ba9d59839be6-kube-api-access-kkdmc\") pod \"isvc-canary-create-predictor-7c7f45c769-97jvx\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:18:59.928402 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:18:59.928314    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:19:00.201750 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:00.201723    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"]
Sep 17 21:19:00.203660 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:19:00.203634    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode070d212_336a_49ac_9325_ba9d59839be6.slice/crio-a2568bb6d97758c6789530637d962d1eeb9f5c1f55f4af40589f5098c9f6235f WatchSource:0}: Error finding container a2568bb6d97758c6789530637d962d1eeb9f5c1f55f4af40589f5098c9f6235f: Status 404 returned error can't find the container with id a2568bb6d97758c6789530637d962d1eeb9f5c1f55f4af40589f5098c9f6235f
Sep 17 21:19:00.267852 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:00.267824    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" event={"ID":"e070d212-336a-49ac-9325-ba9d59839be6","Type":"ContainerStarted","Data":"b491da63a05e900e1fb7e2abb1132f714b23ffeb2cab8959e935f04676faa8e1"}
Sep 17 21:19:00.268002 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:00.267878    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" event={"ID":"e070d212-336a-49ac-9325-ba9d59839be6","Type":"ContainerStarted","Data":"a2568bb6d97758c6789530637d962d1eeb9f5c1f55f4af40589f5098c9f6235f"}
Sep 17 21:19:04.289608 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:04.289573    2571 generic.go:356] "Generic (PLEG): container finished" podID="e070d212-336a-49ac-9325-ba9d59839be6" containerID="b491da63a05e900e1fb7e2abb1132f714b23ffeb2cab8959e935f04676faa8e1" exitCode=0
Sep 17 21:19:04.289970 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:04.289623    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" event={"ID":"e070d212-336a-49ac-9325-ba9d59839be6","Type":"ContainerDied","Data":"b491da63a05e900e1fb7e2abb1132f714b23ffeb2cab8959e935f04676faa8e1"}
Sep 17 21:19:04.290971 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:04.290955    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:19:18.387027 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:18.386936    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" event={"ID":"e070d212-336a-49ac-9325-ba9d59839be6","Type":"ContainerStarted","Data":"5bf8a01097a6752df44527b855129c99806c96c950502abf97aa17ff2758205c"}
Sep 17 21:19:18.387027 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:18.386998    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" event={"ID":"e070d212-336a-49ac-9325-ba9d59839be6","Type":"ContainerStarted","Data":"8412f20d0c3507cc9ef5ff03191eb813c97004a81553fecc0c612da1e62f2706"}
Sep 17 21:19:18.387497 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:18.387321    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:19:18.387497 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:18.387378    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:19:18.389135 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:18.389100    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 21:19:18.407515 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:18.407462    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" podStartSLOduration=1.606201497 podStartE2EDuration="19.407446591s" totalImagesPullingTime="13.826126094s" totalInitContainerRuntime="3.975119s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:18:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:19:18.406403091 +0000 UTC m=+1769.910880692" watchObservedRunningTime="2026-09-17 21:19:18.407446591 +0000 UTC m=+1769.911924187"
Sep 17 21:19:19.392439 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:19.392402    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 21:19:24.397692 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:24.397659    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:19:24.398170 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:24.398145    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 21:19:34.398414 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:34.398329    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 21:19:44.398738 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:44.398699    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 21:19:54.398179 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:19:54.398137    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 21:20:04.398191 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:04.398150    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 21:20:14.398172 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:14.398132    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 21:20:24.398774 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:24.398732    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 21:20:34.398338 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:34.398296    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 21:20:44.399733 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:44.399702    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:20:49.578941 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:49.578898    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"]
Sep 17 21:20:49.579495 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:49.579202    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kserve-container" containerID="cri-o://8412f20d0c3507cc9ef5ff03191eb813c97004a81553fecc0c612da1e62f2706" gracePeriod=30
Sep 17 21:20:49.579495 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:49.579273    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kube-rbac-proxy" containerID="cri-o://5bf8a01097a6752df44527b855129c99806c96c950502abf97aa17ff2758205c" gracePeriod=30
Sep 17 21:20:49.828218 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:49.828183    2571 generic.go:356] "Generic (PLEG): container finished" podID="e070d212-336a-49ac-9325-ba9d59839be6" containerID="5bf8a01097a6752df44527b855129c99806c96c950502abf97aa17ff2758205c" exitCode=2
Sep 17 21:20:49.828387 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:49.828256    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" event={"ID":"e070d212-336a-49ac-9325-ba9d59839be6","Type":"ContainerDied","Data":"5bf8a01097a6752df44527b855129c99806c96c950502abf97aa17ff2758205c"}
Sep 17 21:20:54.393495 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.393447    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 17 21:20:54.398516 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.398486    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 21:20:54.628892 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.628865    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:20:54.721959 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.721867    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e070d212-336a-49ac-9325-ba9d59839be6-proxy-tls\") pod \"e070d212-336a-49ac-9325-ba9d59839be6\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") "
Sep 17 21:20:54.721959 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.721955    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e070d212-336a-49ac-9325-ba9d59839be6-kube-rbac-proxy-sar-config\") pod \"e070d212-336a-49ac-9325-ba9d59839be6\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") "
Sep 17 21:20:54.722182 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.721979    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e070d212-336a-49ac-9325-ba9d59839be6-kserve-provision-location\") pod \"e070d212-336a-49ac-9325-ba9d59839be6\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") "
Sep 17 21:20:54.722182 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.722031    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e070d212-336a-49ac-9325-ba9d59839be6-cabundle-cert\") pod \"e070d212-336a-49ac-9325-ba9d59839be6\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") "
Sep 17 21:20:54.722182 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.722065    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkdmc\" (UniqueName: \"kubernetes.io/projected/e070d212-336a-49ac-9325-ba9d59839be6-kube-api-access-kkdmc\") pod \"e070d212-336a-49ac-9325-ba9d59839be6\" (UID: \"e070d212-336a-49ac-9325-ba9d59839be6\") "
Sep 17 21:20:54.722592 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.722561    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e070d212-336a-49ac-9325-ba9d59839be6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e070d212-336a-49ac-9325-ba9d59839be6" (UID: "e070d212-336a-49ac-9325-ba9d59839be6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:20:54.722786 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.722470    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e070d212-336a-49ac-9325-ba9d59839be6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e070d212-336a-49ac-9325-ba9d59839be6" (UID: "e070d212-336a-49ac-9325-ba9d59839be6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:20:54.723126 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.722646    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e070d212-336a-49ac-9325-ba9d59839be6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e070d212-336a-49ac-9325-ba9d59839be6" (UID: "e070d212-336a-49ac-9325-ba9d59839be6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:20:54.724841 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.724813    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e070d212-336a-49ac-9325-ba9d59839be6-kube-api-access-kkdmc" (OuterVolumeSpecName: "kube-api-access-kkdmc") pod "e070d212-336a-49ac-9325-ba9d59839be6" (UID: "e070d212-336a-49ac-9325-ba9d59839be6"). InnerVolumeSpecName "kube-api-access-kkdmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:20:54.725275 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.725240    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e070d212-336a-49ac-9325-ba9d59839be6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e070d212-336a-49ac-9325-ba9d59839be6" (UID: "e070d212-336a-49ac-9325-ba9d59839be6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:20:54.823526 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.823490    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e070d212-336a-49ac-9325-ba9d59839be6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:20:54.823526 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.823519    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e070d212-336a-49ac-9325-ba9d59839be6-kserve-provision-location\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:20:54.823526 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.823529    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e070d212-336a-49ac-9325-ba9d59839be6-cabundle-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:20:54.823778 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.823538    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkdmc\" (UniqueName: \"kubernetes.io/projected/e070d212-336a-49ac-9325-ba9d59839be6-kube-api-access-kkdmc\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:20:54.823778 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.823571    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e070d212-336a-49ac-9325-ba9d59839be6-proxy-tls\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:20:54.855656 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.855596    2571 generic.go:356] "Generic (PLEG): container finished" podID="e070d212-336a-49ac-9325-ba9d59839be6" containerID="8412f20d0c3507cc9ef5ff03191eb813c97004a81553fecc0c612da1e62f2706" exitCode=0
Sep 17 21:20:54.855892 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.855674    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"
Sep 17 21:20:54.855892 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.855718    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" event={"ID":"e070d212-336a-49ac-9325-ba9d59839be6","Type":"ContainerDied","Data":"8412f20d0c3507cc9ef5ff03191eb813c97004a81553fecc0c612da1e62f2706"}
Sep 17 21:20:54.855892 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.855777    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx" event={"ID":"e070d212-336a-49ac-9325-ba9d59839be6","Type":"ContainerDied","Data":"a2568bb6d97758c6789530637d962d1eeb9f5c1f55f4af40589f5098c9f6235f"}
Sep 17 21:20:54.855892 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.855813    2571 scope.go:117] "RemoveContainer" containerID="5bf8a01097a6752df44527b855129c99806c96c950502abf97aa17ff2758205c"
Sep 17 21:20:54.868606 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.868584    2571 scope.go:117] "RemoveContainer" containerID="8412f20d0c3507cc9ef5ff03191eb813c97004a81553fecc0c612da1e62f2706"
Sep 17 21:20:54.871083 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.871062    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"]
Sep 17 21:20:54.871432 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.871421    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="storage-initializer"
Sep 17 21:20:54.871480 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.871434    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="storage-initializer"
Sep 17 21:20:54.871480 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.871440    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kserve-container"
Sep 17 21:20:54.871480 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.871446    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kserve-container"
Sep 17 21:20:54.871598 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.871479    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kube-rbac-proxy"
Sep 17 21:20:54.871598 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.871488    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kube-rbac-proxy"
Sep 17 21:20:54.871598 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.871583    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kube-rbac-proxy"
Sep 17 21:20:54.871694 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.871597    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e070d212-336a-49ac-9325-ba9d59839be6" containerName="kserve-container"
Sep 17 21:20:54.876440 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.876419    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:54.877954 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.877938    2571 scope.go:117] "RemoveContainer" containerID="b491da63a05e900e1fb7e2abb1132f714b23ffeb2cab8959e935f04676faa8e1"
Sep 17 21:20:54.879128 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.879106    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 21:20:54.879198 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.879174    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:20:54.879260 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.879195    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 21:20:54.879337 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.879317    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:20:54.879397 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.879374    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:20:54.879447 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.879396    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:20:54.879928 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.879892    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:20:54.880070 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.880049    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"]
Sep 17 21:20:54.883454 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.883428    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"]
Sep 17 21:20:54.889478 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.889456    2571 scope.go:117] "RemoveContainer" containerID="5bf8a01097a6752df44527b855129c99806c96c950502abf97aa17ff2758205c"
Sep 17 21:20:54.889803 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:20:54.889785    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf8a01097a6752df44527b855129c99806c96c950502abf97aa17ff2758205c\": container with ID starting with 5bf8a01097a6752df44527b855129c99806c96c950502abf97aa17ff2758205c not found: ID does not exist" containerID="5bf8a01097a6752df44527b855129c99806c96c950502abf97aa17ff2758205c"
Sep 17 21:20:54.889901 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.889816    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf8a01097a6752df44527b855129c99806c96c950502abf97aa17ff2758205c"} err="failed to get container status \"5bf8a01097a6752df44527b855129c99806c96c950502abf97aa17ff2758205c\": rpc error: code = NotFound desc = could not find container \"5bf8a01097a6752df44527b855129c99806c96c950502abf97aa17ff2758205c\": container with ID starting with 5bf8a01097a6752df44527b855129c99806c96c950502abf97aa17ff2758205c not found: ID does not exist"
Sep 17 21:20:54.889901 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.889841    2571 scope.go:117] "RemoveContainer" containerID="8412f20d0c3507cc9ef5ff03191eb813c97004a81553fecc0c612da1e62f2706"
Sep 17 21:20:54.890156 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:20:54.890131    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8412f20d0c3507cc9ef5ff03191eb813c97004a81553fecc0c612da1e62f2706\": container with ID starting with 8412f20d0c3507cc9ef5ff03191eb813c97004a81553fecc0c612da1e62f2706 not found: ID does not exist" containerID="8412f20d0c3507cc9ef5ff03191eb813c97004a81553fecc0c612da1e62f2706"
Sep 17 21:20:54.890217 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.890169    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8412f20d0c3507cc9ef5ff03191eb813c97004a81553fecc0c612da1e62f2706"} err="failed to get container status \"8412f20d0c3507cc9ef5ff03191eb813c97004a81553fecc0c612da1e62f2706\": rpc error: code = NotFound desc = could not find container \"8412f20d0c3507cc9ef5ff03191eb813c97004a81553fecc0c612da1e62f2706\": container with ID starting with 8412f20d0c3507cc9ef5ff03191eb813c97004a81553fecc0c612da1e62f2706 not found: ID does not exist"
Sep 17 21:20:54.890217 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.890193    2571 scope.go:117] "RemoveContainer" containerID="b491da63a05e900e1fb7e2abb1132f714b23ffeb2cab8959e935f04676faa8e1"
Sep 17 21:20:54.890538 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.890515    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-97jvx"]
Sep 17 21:20:54.890745 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:20:54.890580    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b491da63a05e900e1fb7e2abb1132f714b23ffeb2cab8959e935f04676faa8e1\": container with ID starting with b491da63a05e900e1fb7e2abb1132f714b23ffeb2cab8959e935f04676faa8e1 not found: ID does not exist" containerID="b491da63a05e900e1fb7e2abb1132f714b23ffeb2cab8959e935f04676faa8e1"
Sep 17 21:20:54.890745 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:54.890611    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b491da63a05e900e1fb7e2abb1132f714b23ffeb2cab8959e935f04676faa8e1"} err="failed to get container status \"b491da63a05e900e1fb7e2abb1132f714b23ffeb2cab8959e935f04676faa8e1\": rpc error: code = NotFound desc = could not find container \"b491da63a05e900e1fb7e2abb1132f714b23ffeb2cab8959e935f04676faa8e1\": container with ID starting with b491da63a05e900e1fb7e2abb1132f714b23ffeb2cab8959e935f04676faa8e1 not found: ID does not exist"
Sep 17 21:20:55.025470 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.025378    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chx48\" (UniqueName: \"kubernetes.io/projected/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kube-api-access-chx48\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zzh92\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:55.025470 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.025438    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zzh92\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:55.025712 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.025504    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zzh92\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:55.025712 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.025602    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zzh92\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:55.025712 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.025635    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zzh92\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:55.126323 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.126275    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chx48\" (UniqueName: \"kubernetes.io/projected/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kube-api-access-chx48\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zzh92\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:55.126512 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.126335    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zzh92\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:55.126512 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.126388    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zzh92\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:55.126512 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.126438    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zzh92\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:55.126512 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.126463    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zzh92\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:55.126853 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.126833    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zzh92\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:55.127171 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.127149    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zzh92\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:55.127243 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.127179    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zzh92\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:55.128933 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.128914    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zzh92\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:55.137754 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.137734    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chx48\" (UniqueName: \"kubernetes.io/projected/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kube-api-access-chx48\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zzh92\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:55.190436 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.190401    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:20:55.466361 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.466334    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"]
Sep 17 21:20:55.468362 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:20:55.468335    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cf3b90_5d1a_423b_83fc_bdbae5f35d30.slice/crio-0d351ce9325008c43f57ba083ac859319b1abbe1a4509006b9debe37973d315a WatchSource:0}: Error finding container 0d351ce9325008c43f57ba083ac859319b1abbe1a4509006b9debe37973d315a: Status 404 returned error can't find the container with id 0d351ce9325008c43f57ba083ac859319b1abbe1a4509006b9debe37973d315a
Sep 17 21:20:55.863363 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.863323    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" event={"ID":"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30","Type":"ContainerStarted","Data":"1311f995d3eb60026fda7697bf8b971da37dfe4f2eeea4ac1cd451c14c40b465"}
Sep 17 21:20:55.863585 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:55.863384    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" event={"ID":"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30","Type":"ContainerStarted","Data":"0d351ce9325008c43f57ba083ac859319b1abbe1a4509006b9debe37973d315a"}
Sep 17 21:20:56.669810 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:56.669779    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e070d212-336a-49ac-9325-ba9d59839be6" path="/var/lib/kubelet/pods/e070d212-336a-49ac-9325-ba9d59839be6/volumes"
Sep 17 21:20:59.888898 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:59.888857    2571 generic.go:356] "Generic (PLEG): container finished" podID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerID="1311f995d3eb60026fda7697bf8b971da37dfe4f2eeea4ac1cd451c14c40b465" exitCode=0
Sep 17 21:20:59.889277 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:20:59.888922    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" event={"ID":"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30","Type":"ContainerDied","Data":"1311f995d3eb60026fda7697bf8b971da37dfe4f2eeea4ac1cd451c14c40b465"}
Sep 17 21:21:00.895479 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:21:00.895447    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" event={"ID":"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30","Type":"ContainerStarted","Data":"8fce9fe3d0a90f9d809183b3bab31ad67b01b02dc5d65dbaf9061722baf03cf1"}
Sep 17 21:21:00.895876 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:21:00.895507    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" event={"ID":"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30","Type":"ContainerStarted","Data":"6f6b124a6ca2ceaa9a625f6ab88193018d394a225a203aeb1d8a874df97ebf25"}
Sep 17 21:21:00.895876 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:21:00.895800    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:21:00.895987 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:21:00.895971    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:21:00.897253 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:21:00.897226    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:21:00.915531 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:21:00.915488    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" podStartSLOduration=3.012228229 podStartE2EDuration="6.915473838s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.903245609s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:21:00.914311195 +0000 UTC m=+1872.418788791" watchObservedRunningTime="2026-09-17 21:21:00.915473838 +0000 UTC m=+1872.419951430"
Sep 17 21:21:01.900934 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:21:01.900896    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:21:06.905532 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:21:06.905496    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:21:06.906209 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:21:06.906188    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:21:16.906983 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:21:16.906939    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:21:26.906115 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:21:26.906074    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:21:36.906625 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:21:36.906586    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:21:46.906357 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:21:46.906304    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:21:56.907038 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:21:56.906999    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:22:06.906618 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:06.906576    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:22:15.664765 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:15.664723    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:22:25.666516 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:25.666478    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:22:34.890891 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:34.890843    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"]
Sep 17 21:22:34.891293 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:34.891219    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kube-rbac-proxy" containerID="cri-o://8fce9fe3d0a90f9d809183b3bab31ad67b01b02dc5d65dbaf9061722baf03cf1" gracePeriod=30
Sep 17 21:22:34.891379 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:34.891121    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kserve-container" containerID="cri-o://6f6b124a6ca2ceaa9a625f6ab88193018d394a225a203aeb1d8a874df97ebf25" gracePeriod=30
Sep 17 21:22:34.934567 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:34.934521    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"]
Sep 17 21:22:34.938500 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:34.938479    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:34.941279 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:34.941257    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 21:22:34.953693 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:34.953663    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"]
Sep 17 21:22:35.016245 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.016211    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hzq7\" (UniqueName: \"kubernetes.io/projected/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kube-api-access-7hzq7\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:35.016429 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.016268    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2eecd-e0ae-4960-8194-82142a25ed5a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:35.016429 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.016327    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:35.016429 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.016353    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:35.016623 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.016451    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8c2eecd-e0ae-4960-8194-82142a25ed5a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:35.117574 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.117496    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8c2eecd-e0ae-4960-8194-82142a25ed5a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:35.117779 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.117636    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hzq7\" (UniqueName: \"kubernetes.io/projected/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kube-api-access-7hzq7\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:35.117779 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.117683    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2eecd-e0ae-4960-8194-82142a25ed5a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:35.117779 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.117735    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:35.117779 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.117759    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:35.118164 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.118142    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:35.118373 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.118322    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2eecd-e0ae-4960-8194-82142a25ed5a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:35.118373 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.118378    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:35.120272 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.120247    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8c2eecd-e0ae-4960-8194-82142a25ed5a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:35.126820 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.126796    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hzq7\" (UniqueName: \"kubernetes.io/projected/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kube-api-access-7hzq7\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:35.248717 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.248631    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:35.365604 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.365568    2571 generic.go:356] "Generic (PLEG): container finished" podID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerID="8fce9fe3d0a90f9d809183b3bab31ad67b01b02dc5d65dbaf9061722baf03cf1" exitCode=2
Sep 17 21:22:35.365604 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.365591    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" event={"ID":"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30","Type":"ContainerDied","Data":"8fce9fe3d0a90f9d809183b3bab31ad67b01b02dc5d65dbaf9061722baf03cf1"}
Sep 17 21:22:35.530283 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.530258    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"]
Sep 17 21:22:35.532141 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:22:35.532115    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c2eecd_e0ae_4960_8194_82142a25ed5a.slice/crio-6d3dfa3fa6ef5aac6171f7538b2b1ff239d91bd689ef27cfd84859271065b927 WatchSource:0}: Error finding container 6d3dfa3fa6ef5aac6171f7538b2b1ff239d91bd689ef27cfd84859271065b927: Status 404 returned error can't find the container with id 6d3dfa3fa6ef5aac6171f7538b2b1ff239d91bd689ef27cfd84859271065b927
Sep 17 21:22:35.665078 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:35.665035    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 21:22:36.372123 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:36.372089    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" event={"ID":"f8c2eecd-e0ae-4960-8194-82142a25ed5a","Type":"ContainerStarted","Data":"bd3ef246f672ee014ebb7b31555690023973ebb336bdcce1ae0386f46cd8eb96"}
Sep 17 21:22:36.372495 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:36.372144    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" event={"ID":"f8c2eecd-e0ae-4960-8194-82142a25ed5a","Type":"ContainerStarted","Data":"6d3dfa3fa6ef5aac6171f7538b2b1ff239d91bd689ef27cfd84859271065b927"}
Sep 17 21:22:36.901466 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:36.901425    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 21:22:39.389667 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:39.389630    2571 generic.go:356] "Generic (PLEG): container finished" podID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerID="bd3ef246f672ee014ebb7b31555690023973ebb336bdcce1ae0386f46cd8eb96" exitCode=0
Sep 17 21:22:39.390106 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:39.389676    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" event={"ID":"f8c2eecd-e0ae-4960-8194-82142a25ed5a","Type":"ContainerDied","Data":"bd3ef246f672ee014ebb7b31555690023973ebb336bdcce1ae0386f46cd8eb96"}
Sep 17 21:22:40.043214 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.043188    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:22:40.166430 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.166325    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-proxy-tls\") pod \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") "
Sep 17 21:22:40.166430 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.166372    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-cabundle-cert\") pod \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") "
Sep 17 21:22:40.166711 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.166448    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kserve-provision-location\") pod \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") "
Sep 17 21:22:40.166711 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.166512    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chx48\" (UniqueName: \"kubernetes.io/projected/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kube-api-access-chx48\") pod \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") "
Sep 17 21:22:40.166711 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.166590    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kube-rbac-proxy-sar-config\") pod \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\" (UID: \"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30\") "
Sep 17 21:22:40.166863 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.166814    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" (UID: "f5cf3b90-5d1a-423b-83fc-bdbae5f35d30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:22:40.166863 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.166822    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" (UID: "f5cf3b90-5d1a-423b-83fc-bdbae5f35d30"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:22:40.167003 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.166981    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" (UID: "f5cf3b90-5d1a-423b-83fc-bdbae5f35d30"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:22:40.167061 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.167009    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-cabundle-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:22:40.167061 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.167024    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kserve-provision-location\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:22:40.168706 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.168686    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" (UID: "f5cf3b90-5d1a-423b-83fc-bdbae5f35d30"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:22:40.168873 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.168858    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kube-api-access-chx48" (OuterVolumeSpecName: "kube-api-access-chx48") pod "f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" (UID: "f5cf3b90-5d1a-423b-83fc-bdbae5f35d30"). InnerVolumeSpecName "kube-api-access-chx48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:22:40.268344 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.268310    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:22:40.268344 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.268342    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-proxy-tls\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:22:40.268564 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.268356    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chx48\" (UniqueName: \"kubernetes.io/projected/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30-kube-api-access-chx48\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:22:40.397353 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.397319    2571 generic.go:356] "Generic (PLEG): container finished" podID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerID="6f6b124a6ca2ceaa9a625f6ab88193018d394a225a203aeb1d8a874df97ebf25" exitCode=0
Sep 17 21:22:40.397797 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.397392    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"
Sep 17 21:22:40.397797 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.397408    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" event={"ID":"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30","Type":"ContainerDied","Data":"6f6b124a6ca2ceaa9a625f6ab88193018d394a225a203aeb1d8a874df97ebf25"}
Sep 17 21:22:40.397797 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.397494    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92" event={"ID":"f5cf3b90-5d1a-423b-83fc-bdbae5f35d30","Type":"ContainerDied","Data":"0d351ce9325008c43f57ba083ac859319b1abbe1a4509006b9debe37973d315a"}
Sep 17 21:22:40.397797 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.397522    2571 scope.go:117] "RemoveContainer" containerID="8fce9fe3d0a90f9d809183b3bab31ad67b01b02dc5d65dbaf9061722baf03cf1"
Sep 17 21:22:40.399974 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.399946    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" event={"ID":"f8c2eecd-e0ae-4960-8194-82142a25ed5a","Type":"ContainerStarted","Data":"325f59e3032d0cf09c864eef7f1705cc270b2eb7e15fbb6583f4a2a0418cc8e2"}
Sep 17 21:22:40.400089 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.400022    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" event={"ID":"f8c2eecd-e0ae-4960-8194-82142a25ed5a","Type":"ContainerStarted","Data":"fca70c8e5f6ac0afe29b2d52c38f0a08f6f12c782f781292e5d44aaa79a471e6"}
Sep 17 21:22:40.400305 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.400289    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:40.407512 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.407497    2571 scope.go:117] "RemoveContainer" containerID="6f6b124a6ca2ceaa9a625f6ab88193018d394a225a203aeb1d8a874df97ebf25"
Sep 17 21:22:40.415599 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.415581    2571 scope.go:117] "RemoveContainer" containerID="1311f995d3eb60026fda7697bf8b971da37dfe4f2eeea4ac1cd451c14c40b465"
Sep 17 21:22:40.422404 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.422334    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" podStartSLOduration=2.738932992 podStartE2EDuration="6.42232149s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.683388498s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:22:40.420384828 +0000 UTC m=+1971.924862427" watchObservedRunningTime="2026-09-17 21:22:40.42232149 +0000 UTC m=+1971.926799074"
Sep 17 21:22:40.423966 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.423949    2571 scope.go:117] "RemoveContainer" containerID="8fce9fe3d0a90f9d809183b3bab31ad67b01b02dc5d65dbaf9061722baf03cf1"
Sep 17 21:22:40.424272 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:22:40.424250    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fce9fe3d0a90f9d809183b3bab31ad67b01b02dc5d65dbaf9061722baf03cf1\": container with ID starting with 8fce9fe3d0a90f9d809183b3bab31ad67b01b02dc5d65dbaf9061722baf03cf1 not found: ID does not exist" containerID="8fce9fe3d0a90f9d809183b3bab31ad67b01b02dc5d65dbaf9061722baf03cf1"
Sep 17 21:22:40.424365 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.424280    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fce9fe3d0a90f9d809183b3bab31ad67b01b02dc5d65dbaf9061722baf03cf1"} err="failed to get container status \"8fce9fe3d0a90f9d809183b3bab31ad67b01b02dc5d65dbaf9061722baf03cf1\": rpc error: code = NotFound desc = could not find container \"8fce9fe3d0a90f9d809183b3bab31ad67b01b02dc5d65dbaf9061722baf03cf1\": container with ID starting with 8fce9fe3d0a90f9d809183b3bab31ad67b01b02dc5d65dbaf9061722baf03cf1 not found: ID does not exist"
Sep 17 21:22:40.424365 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.424298    2571 scope.go:117] "RemoveContainer" containerID="6f6b124a6ca2ceaa9a625f6ab88193018d394a225a203aeb1d8a874df97ebf25"
Sep 17 21:22:40.424623 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:22:40.424605    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f6b124a6ca2ceaa9a625f6ab88193018d394a225a203aeb1d8a874df97ebf25\": container with ID starting with 6f6b124a6ca2ceaa9a625f6ab88193018d394a225a203aeb1d8a874df97ebf25 not found: ID does not exist" containerID="6f6b124a6ca2ceaa9a625f6ab88193018d394a225a203aeb1d8a874df97ebf25"
Sep 17 21:22:40.424692 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.424631    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f6b124a6ca2ceaa9a625f6ab88193018d394a225a203aeb1d8a874df97ebf25"} err="failed to get container status \"6f6b124a6ca2ceaa9a625f6ab88193018d394a225a203aeb1d8a874df97ebf25\": rpc error: code = NotFound desc = could not find container \"6f6b124a6ca2ceaa9a625f6ab88193018d394a225a203aeb1d8a874df97ebf25\": container with ID starting with 6f6b124a6ca2ceaa9a625f6ab88193018d394a225a203aeb1d8a874df97ebf25 not found: ID does not exist"
Sep 17 21:22:40.424692 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.424646    2571 scope.go:117] "RemoveContainer" containerID="1311f995d3eb60026fda7697bf8b971da37dfe4f2eeea4ac1cd451c14c40b465"
Sep 17 21:22:40.424894 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:22:40.424878    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1311f995d3eb60026fda7697bf8b971da37dfe4f2eeea4ac1cd451c14c40b465\": container with ID starting with 1311f995d3eb60026fda7697bf8b971da37dfe4f2eeea4ac1cd451c14c40b465 not found: ID does not exist" containerID="1311f995d3eb60026fda7697bf8b971da37dfe4f2eeea4ac1cd451c14c40b465"
Sep 17 21:22:40.424931 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.424898    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1311f995d3eb60026fda7697bf8b971da37dfe4f2eeea4ac1cd451c14c40b465"} err="failed to get container status \"1311f995d3eb60026fda7697bf8b971da37dfe4f2eeea4ac1cd451c14c40b465\": rpc error: code = NotFound desc = could not find container \"1311f995d3eb60026fda7697bf8b971da37dfe4f2eeea4ac1cd451c14c40b465\": container with ID starting with 1311f995d3eb60026fda7697bf8b971da37dfe4f2eeea4ac1cd451c14c40b465 not found: ID does not exist"
Sep 17 21:22:40.434434 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.434411    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"]
Sep 17 21:22:40.438525 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.438504    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zzh92"]
Sep 17 21:22:40.671681 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:40.671651    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" path="/var/lib/kubelet/pods/f5cf3b90-5d1a-423b-83fc-bdbae5f35d30/volumes"
Sep 17 21:22:41.406097 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:41.406068    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:41.407648 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:41.407620    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 21:22:42.411403 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:42.411361    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 21:22:47.416430 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:47.416403    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:22:47.416958 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:47.416931    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 21:22:57.416980 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:22:57.416936    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 21:23:07.417753 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:23:07.417716    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 21:23:17.417448 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:23:17.417409    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 21:23:27.417249 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:23:27.417212    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 21:23:37.417463 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:23:37.417418    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 21:23:47.417564 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:23:47.417513    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 21:23:57.418274 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:23:57.418239    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:24:05.041750 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:05.041526    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"]
Sep 17 21:24:05.042188 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:05.042107    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kserve-container" containerID="cri-o://fca70c8e5f6ac0afe29b2d52c38f0a08f6f12c782f781292e5d44aaa79a471e6" gracePeriod=30
Sep 17 21:24:05.042490 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:05.042317    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kube-rbac-proxy" containerID="cri-o://325f59e3032d0cf09c864eef7f1705cc270b2eb7e15fbb6583f4a2a0418cc8e2" gracePeriod=30
Sep 17 21:24:05.811605 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:05.811571    2571 generic.go:356] "Generic (PLEG): container finished" podID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerID="325f59e3032d0cf09c864eef7f1705cc270b2eb7e15fbb6583f4a2a0418cc8e2" exitCode=2
Sep 17 21:24:05.811605 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:05.811589    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" event={"ID":"f8c2eecd-e0ae-4960-8194-82142a25ed5a","Type":"ContainerDied","Data":"325f59e3032d0cf09c864eef7f1705cc270b2eb7e15fbb6583f4a2a0418cc8e2"}
Sep 17 21:24:07.411617 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:07.411573    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 17 21:24:07.416926 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:07.416898    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 21:24:10.202139 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.202106    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:24:10.248124 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248048    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"]
Sep 17 21:24:10.248419 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248407    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kserve-container"
Sep 17 21:24:10.248463 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248421    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kserve-container"
Sep 17 21:24:10.248463 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248431    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="storage-initializer"
Sep 17 21:24:10.248463 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248437    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="storage-initializer"
Sep 17 21:24:10.248463 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248450    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kube-rbac-proxy"
Sep 17 21:24:10.248463 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248456    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kube-rbac-proxy"
Sep 17 21:24:10.248665 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248469    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="storage-initializer"
Sep 17 21:24:10.248665 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248475    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="storage-initializer"
Sep 17 21:24:10.248665 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248481    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kube-rbac-proxy"
Sep 17 21:24:10.248665 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248486    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kube-rbac-proxy"
Sep 17 21:24:10.248665 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248499    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kserve-container"
Sep 17 21:24:10.248665 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248512    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kserve-container"
Sep 17 21:24:10.248665 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248582    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kserve-container"
Sep 17 21:24:10.248665 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248592    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kserve-container"
Sep 17 21:24:10.248665 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248626    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerName="kube-rbac-proxy"
Sep 17 21:24:10.248665 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.248633    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5cf3b90-5d1a-423b-83fc-bdbae5f35d30" containerName="kube-rbac-proxy"
Sep 17 21:24:10.251785 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.251762    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.254876 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.254848    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 21:24:10.255165 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.255145    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 21:24:10.266675 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.266647    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"]
Sep 17 21:24:10.302536 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.302502    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hzq7\" (UniqueName: \"kubernetes.io/projected/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kube-api-access-7hzq7\") pod \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") "
Sep 17 21:24:10.302728 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.302603    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kube-rbac-proxy-sar-config\") pod \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") "
Sep 17 21:24:10.302728 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.302630    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kserve-provision-location\") pod \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") "
Sep 17 21:24:10.302728 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.302683    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8c2eecd-e0ae-4960-8194-82142a25ed5a-proxy-tls\") pod \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") "
Sep 17 21:24:10.302898 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.302819    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2eecd-e0ae-4960-8194-82142a25ed5a-cabundle-cert\") pod \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\" (UID: \"f8c2eecd-e0ae-4960-8194-82142a25ed5a\") "
Sep 17 21:24:10.303089 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.302998    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8c2eecd-e0ae-4960-8194-82142a25ed5a" (UID: "f8c2eecd-e0ae-4960-8194-82142a25ed5a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:24:10.303089 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.303099    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f8c2eecd-e0ae-4960-8194-82142a25ed5a" (UID: "f8c2eecd-e0ae-4960-8194-82142a25ed5a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:24:10.303708 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.303138    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c2eecd-e0ae-4960-8194-82142a25ed5a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f8c2eecd-e0ae-4960-8194-82142a25ed5a" (UID: "f8c2eecd-e0ae-4960-8194-82142a25ed5a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:24:10.303708 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.303154    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kserve-provision-location\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:24:10.305079 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.305055    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kube-api-access-7hzq7" (OuterVolumeSpecName: "kube-api-access-7hzq7") pod "f8c2eecd-e0ae-4960-8194-82142a25ed5a" (UID: "f8c2eecd-e0ae-4960-8194-82142a25ed5a"). InnerVolumeSpecName "kube-api-access-7hzq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:24:10.305155 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.305106    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c2eecd-e0ae-4960-8194-82142a25ed5a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f8c2eecd-e0ae-4960-8194-82142a25ed5a" (UID: "f8c2eecd-e0ae-4960-8194-82142a25ed5a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:24:10.404239 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.404204    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d871933-528e-4880-a65c-7337151e63b3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-q9wv2\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.404436 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.404264    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d871933-528e-4880-a65c-7337151e63b3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-q9wv2\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.404436 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.404349    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d871933-528e-4880-a65c-7337151e63b3-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-q9wv2\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.404436 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.404385    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s2rc\" (UniqueName: \"kubernetes.io/projected/6d871933-528e-4880-a65c-7337151e63b3-kube-api-access-4s2rc\") pod \"isvc-canary-rollback-predictor-97977b6b5-q9wv2\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.404436 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.404403    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d871933-528e-4880-a65c-7337151e63b3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-q9wv2\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.404627 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.404455    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8c2eecd-e0ae-4960-8194-82142a25ed5a-proxy-tls\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:24:10.404627 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.404466    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2eecd-e0ae-4960-8194-82142a25ed5a-cabundle-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:24:10.404627 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.404475    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hzq7\" (UniqueName: \"kubernetes.io/projected/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kube-api-access-7hzq7\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:24:10.404627 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.404484    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8c2eecd-e0ae-4960-8194-82142a25ed5a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:24:10.505217 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.505064    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d871933-528e-4880-a65c-7337151e63b3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-q9wv2\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.505217 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.505130    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d871933-528e-4880-a65c-7337151e63b3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-q9wv2\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.505217 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.505169    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d871933-528e-4880-a65c-7337151e63b3-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-q9wv2\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.505505 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.505222    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s2rc\" (UniqueName: \"kubernetes.io/projected/6d871933-528e-4880-a65c-7337151e63b3-kube-api-access-4s2rc\") pod \"isvc-canary-rollback-predictor-97977b6b5-q9wv2\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.505505 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.505258    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d871933-528e-4880-a65c-7337151e63b3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-q9wv2\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.505655 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.505626    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d871933-528e-4880-a65c-7337151e63b3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-q9wv2\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.505962 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.505937    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d871933-528e-4880-a65c-7337151e63b3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-q9wv2\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.506146 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.506129    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d871933-528e-4880-a65c-7337151e63b3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-q9wv2\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.508171 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.508151    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d871933-528e-4880-a65c-7337151e63b3-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-q9wv2\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.513489 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.513469    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s2rc\" (UniqueName: \"kubernetes.io/projected/6d871933-528e-4880-a65c-7337151e63b3-kube-api-access-4s2rc\") pod \"isvc-canary-rollback-predictor-97977b6b5-q9wv2\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.568381 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.568343    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:10.837413 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.837380    2571 generic.go:356] "Generic (PLEG): container finished" podID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" containerID="fca70c8e5f6ac0afe29b2d52c38f0a08f6f12c782f781292e5d44aaa79a471e6" exitCode=0
Sep 17 21:24:10.837638 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.837454    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" event={"ID":"f8c2eecd-e0ae-4960-8194-82142a25ed5a","Type":"ContainerDied","Data":"fca70c8e5f6ac0afe29b2d52c38f0a08f6f12c782f781292e5d44aaa79a471e6"}
Sep 17 21:24:10.837638 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.837514    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb" event={"ID":"f8c2eecd-e0ae-4960-8194-82142a25ed5a","Type":"ContainerDied","Data":"6d3dfa3fa6ef5aac6171f7538b2b1ff239d91bd689ef27cfd84859271065b927"}
Sep 17 21:24:10.837638 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.837540    2571 scope.go:117] "RemoveContainer" containerID="325f59e3032d0cf09c864eef7f1705cc270b2eb7e15fbb6583f4a2a0418cc8e2"
Sep 17 21:24:10.837638 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.837465    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"
Sep 17 21:24:10.844761 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.844728    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"]
Sep 17 21:24:10.847584 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.847564    2571 scope.go:117] "RemoveContainer" containerID="fca70c8e5f6ac0afe29b2d52c38f0a08f6f12c782f781292e5d44aaa79a471e6"
Sep 17 21:24:10.849478 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:24:10.849452    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d871933_528e_4880_a65c_7337151e63b3.slice/crio-124ac3432edad2925160548a4d05cec2d6767417c9cdc41aa83e2721afd924d3 WatchSource:0}: Error finding container 124ac3432edad2925160548a4d05cec2d6767417c9cdc41aa83e2721afd924d3: Status 404 returned error can't find the container with id 124ac3432edad2925160548a4d05cec2d6767417c9cdc41aa83e2721afd924d3
Sep 17 21:24:10.856259 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.856243    2571 scope.go:117] "RemoveContainer" containerID="bd3ef246f672ee014ebb7b31555690023973ebb336bdcce1ae0386f46cd8eb96"
Sep 17 21:24:10.859518 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.859493    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"]
Sep 17 21:24:10.863565 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.863525    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-nn6tb"]
Sep 17 21:24:10.865947 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.865924    2571 scope.go:117] "RemoveContainer" containerID="325f59e3032d0cf09c864eef7f1705cc270b2eb7e15fbb6583f4a2a0418cc8e2"
Sep 17 21:24:10.866274 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:24:10.866240    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325f59e3032d0cf09c864eef7f1705cc270b2eb7e15fbb6583f4a2a0418cc8e2\": container with ID starting with 325f59e3032d0cf09c864eef7f1705cc270b2eb7e15fbb6583f4a2a0418cc8e2 not found: ID does not exist" containerID="325f59e3032d0cf09c864eef7f1705cc270b2eb7e15fbb6583f4a2a0418cc8e2"
Sep 17 21:24:10.866317 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.866274    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325f59e3032d0cf09c864eef7f1705cc270b2eb7e15fbb6583f4a2a0418cc8e2"} err="failed to get container status \"325f59e3032d0cf09c864eef7f1705cc270b2eb7e15fbb6583f4a2a0418cc8e2\": rpc error: code = NotFound desc = could not find container \"325f59e3032d0cf09c864eef7f1705cc270b2eb7e15fbb6583f4a2a0418cc8e2\": container with ID starting with 325f59e3032d0cf09c864eef7f1705cc270b2eb7e15fbb6583f4a2a0418cc8e2 not found: ID does not exist"
Sep 17 21:24:10.866317 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.866292    2571 scope.go:117] "RemoveContainer" containerID="fca70c8e5f6ac0afe29b2d52c38f0a08f6f12c782f781292e5d44aaa79a471e6"
Sep 17 21:24:10.866571 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:24:10.866529    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca70c8e5f6ac0afe29b2d52c38f0a08f6f12c782f781292e5d44aaa79a471e6\": container with ID starting with fca70c8e5f6ac0afe29b2d52c38f0a08f6f12c782f781292e5d44aaa79a471e6 not found: ID does not exist" containerID="fca70c8e5f6ac0afe29b2d52c38f0a08f6f12c782f781292e5d44aaa79a471e6"
Sep 17 21:24:10.866648 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.866574    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca70c8e5f6ac0afe29b2d52c38f0a08f6f12c782f781292e5d44aaa79a471e6"} err="failed to get container status \"fca70c8e5f6ac0afe29b2d52c38f0a08f6f12c782f781292e5d44aaa79a471e6\": rpc error: code = NotFound desc = could not find container \"fca70c8e5f6ac0afe29b2d52c38f0a08f6f12c782f781292e5d44aaa79a471e6\": container with ID starting with fca70c8e5f6ac0afe29b2d52c38f0a08f6f12c782f781292e5d44aaa79a471e6 not found: ID does not exist"
Sep 17 21:24:10.866648 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.866589    2571 scope.go:117] "RemoveContainer" containerID="bd3ef246f672ee014ebb7b31555690023973ebb336bdcce1ae0386f46cd8eb96"
Sep 17 21:24:10.866869 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:24:10.866850    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd3ef246f672ee014ebb7b31555690023973ebb336bdcce1ae0386f46cd8eb96\": container with ID starting with bd3ef246f672ee014ebb7b31555690023973ebb336bdcce1ae0386f46cd8eb96 not found: ID does not exist" containerID="bd3ef246f672ee014ebb7b31555690023973ebb336bdcce1ae0386f46cd8eb96"
Sep 17 21:24:10.866935 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:10.866879    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd3ef246f672ee014ebb7b31555690023973ebb336bdcce1ae0386f46cd8eb96"} err="failed to get container status \"bd3ef246f672ee014ebb7b31555690023973ebb336bdcce1ae0386f46cd8eb96\": rpc error: code = NotFound desc = could not find container \"bd3ef246f672ee014ebb7b31555690023973ebb336bdcce1ae0386f46cd8eb96\": container with ID starting with bd3ef246f672ee014ebb7b31555690023973ebb336bdcce1ae0386f46cd8eb96 not found: ID does not exist"
Sep 17 21:24:11.844587 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:11.844535    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" event={"ID":"6d871933-528e-4880-a65c-7337151e63b3","Type":"ContainerStarted","Data":"ed031e90d50910824ce414e1843c8f469be1bbde1a237e6f6e49186e10fb1b51"}
Sep 17 21:24:11.845053 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:11.844615    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" event={"ID":"6d871933-528e-4880-a65c-7337151e63b3","Type":"ContainerStarted","Data":"124ac3432edad2925160548a4d05cec2d6767417c9cdc41aa83e2721afd924d3"}
Sep 17 21:24:12.670438 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:12.670403    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8c2eecd-e0ae-4960-8194-82142a25ed5a" path="/var/lib/kubelet/pods/f8c2eecd-e0ae-4960-8194-82142a25ed5a/volumes"
Sep 17 21:24:14.862502 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:14.862473    2571 generic.go:356] "Generic (PLEG): container finished" podID="6d871933-528e-4880-a65c-7337151e63b3" containerID="ed031e90d50910824ce414e1843c8f469be1bbde1a237e6f6e49186e10fb1b51" exitCode=0
Sep 17 21:24:14.862900 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:14.862561    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" event={"ID":"6d871933-528e-4880-a65c-7337151e63b3","Type":"ContainerDied","Data":"ed031e90d50910824ce414e1843c8f469be1bbde1a237e6f6e49186e10fb1b51"}
Sep 17 21:24:15.869941 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:15.869907    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" event={"ID":"6d871933-528e-4880-a65c-7337151e63b3","Type":"ContainerStarted","Data":"afd9b8ae4bf0703dd1861061d76ff6cb179d8e2ebd7e2c1b8bc8c967b4031670"}
Sep 17 21:24:15.870365 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:15.869991    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" event={"ID":"6d871933-528e-4880-a65c-7337151e63b3","Type":"ContainerStarted","Data":"8624785ba415ecd374c9259e5b916c8ae8f29bd13205a1b52decc94c29afc772"}
Sep 17 21:24:15.870365 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:15.870171    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:15.892097 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:15.892042    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" podStartSLOduration=2.071219933 podStartE2EDuration="5.892027672s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.820807739s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:24:15.889997449 +0000 UTC m=+2067.394475045" watchObservedRunningTime="2026-09-17 21:24:15.892027672 +0000 UTC m=+2067.396505337"
Sep 17 21:24:16.874322 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:16.874285    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:16.875679 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:16.875648    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 21:24:17.879436 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:17.879394    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 21:24:22.884274 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:22.884242    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:24:22.884830 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:22.884805    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 21:24:32.884772 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:32.884733    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 21:24:42.885252 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:42.885203    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 21:24:52.884831 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:24:52.884786    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 21:25:02.885648 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:25:02.885608    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 21:25:12.885359 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:25:12.885317    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 21:25:22.885296 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:25:22.885248    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 21:25:26.664369 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:25:26.664329    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 21:25:36.671479 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:25:36.671400    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:27:30.442654 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:30.442543    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"]
Sep 17 21:27:30.443194 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:30.442834    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kserve-container" containerID="cri-o://8624785ba415ecd374c9259e5b916c8ae8f29bd13205a1b52decc94c29afc772" gracePeriod=30
Sep 17 21:27:30.443194 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:30.442861    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kube-rbac-proxy" containerID="cri-o://afd9b8ae4bf0703dd1861061d76ff6cb179d8e2ebd7e2c1b8bc8c967b4031670" gracePeriod=30
Sep 17 21:27:30.846396 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:30.846362    2571 generic.go:356] "Generic (PLEG): container finished" podID="6d871933-528e-4880-a65c-7337151e63b3" containerID="afd9b8ae4bf0703dd1861061d76ff6cb179d8e2ebd7e2c1b8bc8c967b4031670" exitCode=2
Sep 17 21:27:30.846596 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:30.846441    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" event={"ID":"6d871933-528e-4880-a65c-7337151e63b3","Type":"ContainerDied","Data":"afd9b8ae4bf0703dd1861061d76ff6cb179d8e2ebd7e2c1b8bc8c967b4031670"}
Sep 17 21:27:32.880800 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:32.880757    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 17 21:27:35.387368 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.387342    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:27:35.494367 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.494271    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d871933-528e-4880-a65c-7337151e63b3-cabundle-cert\") pod \"6d871933-528e-4880-a65c-7337151e63b3\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") "
Sep 17 21:27:35.494367 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.494358    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d871933-528e-4880-a65c-7337151e63b3-kserve-provision-location\") pod \"6d871933-528e-4880-a65c-7337151e63b3\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") "
Sep 17 21:27:35.494623 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.494381    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d871933-528e-4880-a65c-7337151e63b3-kube-rbac-proxy-sar-config\") pod \"6d871933-528e-4880-a65c-7337151e63b3\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") "
Sep 17 21:27:35.494623 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.494401    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d871933-528e-4880-a65c-7337151e63b3-proxy-tls\") pod \"6d871933-528e-4880-a65c-7337151e63b3\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") "
Sep 17 21:27:35.494623 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.494456    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s2rc\" (UniqueName: \"kubernetes.io/projected/6d871933-528e-4880-a65c-7337151e63b3-kube-api-access-4s2rc\") pod \"6d871933-528e-4880-a65c-7337151e63b3\" (UID: \"6d871933-528e-4880-a65c-7337151e63b3\") "
Sep 17 21:27:35.494845 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.494686    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d871933-528e-4880-a65c-7337151e63b3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6d871933-528e-4880-a65c-7337151e63b3" (UID: "6d871933-528e-4880-a65c-7337151e63b3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:27:35.494845 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.494731    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d871933-528e-4880-a65c-7337151e63b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6d871933-528e-4880-a65c-7337151e63b3" (UID: "6d871933-528e-4880-a65c-7337151e63b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:27:35.494845 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.494738    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d871933-528e-4880-a65c-7337151e63b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6d871933-528e-4880-a65c-7337151e63b3" (UID: "6d871933-528e-4880-a65c-7337151e63b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:27:35.496851 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.496822    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d871933-528e-4880-a65c-7337151e63b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d871933-528e-4880-a65c-7337151e63b3" (UID: "6d871933-528e-4880-a65c-7337151e63b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:27:35.496976 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.496812    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d871933-528e-4880-a65c-7337151e63b3-kube-api-access-4s2rc" (OuterVolumeSpecName: "kube-api-access-4s2rc") pod "6d871933-528e-4880-a65c-7337151e63b3" (UID: "6d871933-528e-4880-a65c-7337151e63b3"). InnerVolumeSpecName "kube-api-access-4s2rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:27:35.595379 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.595343    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d871933-528e-4880-a65c-7337151e63b3-proxy-tls\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:27:35.595379 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.595371    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s2rc\" (UniqueName: \"kubernetes.io/projected/6d871933-528e-4880-a65c-7337151e63b3-kube-api-access-4s2rc\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:27:35.595379 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.595381    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d871933-528e-4880-a65c-7337151e63b3-cabundle-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:27:35.595633 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.595391    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d871933-528e-4880-a65c-7337151e63b3-kserve-provision-location\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:27:35.595633 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.595401    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d871933-528e-4880-a65c-7337151e63b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:27:35.706648 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.706610    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"]
Sep 17 21:27:35.706993 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.706981    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kserve-container"
Sep 17 21:27:35.707041 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.706995    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kserve-container"
Sep 17 21:27:35.707041 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.707006    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kube-rbac-proxy"
Sep 17 21:27:35.707041 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.707012    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kube-rbac-proxy"
Sep 17 21:27:35.707041 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.707030    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="storage-initializer"
Sep 17 21:27:35.707041 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.707036    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="storage-initializer"
Sep 17 21:27:35.707198 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.707095    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kserve-container"
Sep 17 21:27:35.707198 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.707102    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d871933-528e-4880-a65c-7337151e63b3" containerName="kube-rbac-proxy"
Sep 17 21:27:35.710599 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.710577    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:35.712987 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.712965    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 21:27:35.713098 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.713004    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 21:27:35.722321 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.722295    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"]
Sep 17 21:27:35.797882 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.797795    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a54a1b-012b-46b2-9878-97c2b1973e9c-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-nwcrt\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:35.797882 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.797863    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-nwcrt\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:35.798097 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.797896    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-nwcrt\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:35.798097 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.797927    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5a54a1b-012b-46b2-9878-97c2b1973e9c-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-nwcrt\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:35.798097 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.798008    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9jhp\" (UniqueName: \"kubernetes.io/projected/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kube-api-access-p9jhp\") pod \"isvc-canary-stop-predictor-55656fdf77-nwcrt\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:35.872634 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.872601    2571 generic.go:356] "Generic (PLEG): container finished" podID="6d871933-528e-4880-a65c-7337151e63b3" containerID="8624785ba415ecd374c9259e5b916c8ae8f29bd13205a1b52decc94c29afc772" exitCode=0
Sep 17 21:27:35.872797 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.872667    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"
Sep 17 21:27:35.872797 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.872684    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" event={"ID":"6d871933-528e-4880-a65c-7337151e63b3","Type":"ContainerDied","Data":"8624785ba415ecd374c9259e5b916c8ae8f29bd13205a1b52decc94c29afc772"}
Sep 17 21:27:35.872797 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.872751    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2" event={"ID":"6d871933-528e-4880-a65c-7337151e63b3","Type":"ContainerDied","Data":"124ac3432edad2925160548a4d05cec2d6767417c9cdc41aa83e2721afd924d3"}
Sep 17 21:27:35.872797 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.872779    2571 scope.go:117] "RemoveContainer" containerID="afd9b8ae4bf0703dd1861061d76ff6cb179d8e2ebd7e2c1b8bc8c967b4031670"
Sep 17 21:27:35.882989 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.882970    2571 scope.go:117] "RemoveContainer" containerID="8624785ba415ecd374c9259e5b916c8ae8f29bd13205a1b52decc94c29afc772"
Sep 17 21:27:35.892159 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.892144    2571 scope.go:117] "RemoveContainer" containerID="ed031e90d50910824ce414e1843c8f469be1bbde1a237e6f6e49186e10fb1b51"
Sep 17 21:27:35.898483 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.898458    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-nwcrt\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:35.898621 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.898502    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-nwcrt\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:35.898621 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.898536    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5a54a1b-012b-46b2-9878-97c2b1973e9c-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-nwcrt\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:35.898754 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.898676    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9jhp\" (UniqueName: \"kubernetes.io/projected/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kube-api-access-p9jhp\") pod \"isvc-canary-stop-predictor-55656fdf77-nwcrt\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:35.898812 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.898771    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a54a1b-012b-46b2-9878-97c2b1973e9c-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-nwcrt\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:35.898907 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.898884    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-nwcrt\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:35.899443 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.899422    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a54a1b-012b-46b2-9878-97c2b1973e9c-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-nwcrt\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:35.899659 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.899632    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-nwcrt\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:35.900370 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.900349    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"]
Sep 17 21:27:35.902239 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.902215    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5a54a1b-012b-46b2-9878-97c2b1973e9c-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-nwcrt\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:35.903024 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.903003    2571 scope.go:117] "RemoveContainer" containerID="afd9b8ae4bf0703dd1861061d76ff6cb179d8e2ebd7e2c1b8bc8c967b4031670"
Sep 17 21:27:35.903192 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.903168    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-q9wv2"]
Sep 17 21:27:35.903351 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:27:35.903333    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd9b8ae4bf0703dd1861061d76ff6cb179d8e2ebd7e2c1b8bc8c967b4031670\": container with ID starting with afd9b8ae4bf0703dd1861061d76ff6cb179d8e2ebd7e2c1b8bc8c967b4031670 not found: ID does not exist" containerID="afd9b8ae4bf0703dd1861061d76ff6cb179d8e2ebd7e2c1b8bc8c967b4031670"
Sep 17 21:27:35.903411 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.903362    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd9b8ae4bf0703dd1861061d76ff6cb179d8e2ebd7e2c1b8bc8c967b4031670"} err="failed to get container status \"afd9b8ae4bf0703dd1861061d76ff6cb179d8e2ebd7e2c1b8bc8c967b4031670\": rpc error: code = NotFound desc = could not find container \"afd9b8ae4bf0703dd1861061d76ff6cb179d8e2ebd7e2c1b8bc8c967b4031670\": container with ID starting with afd9b8ae4bf0703dd1861061d76ff6cb179d8e2ebd7e2c1b8bc8c967b4031670 not found: ID does not exist"
Sep 17 21:27:35.903411 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.903379    2571 scope.go:117] "RemoveContainer" containerID="8624785ba415ecd374c9259e5b916c8ae8f29bd13205a1b52decc94c29afc772"
Sep 17 21:27:35.903648 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:27:35.903624    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8624785ba415ecd374c9259e5b916c8ae8f29bd13205a1b52decc94c29afc772\": container with ID starting with 8624785ba415ecd374c9259e5b916c8ae8f29bd13205a1b52decc94c29afc772 not found: ID does not exist" containerID="8624785ba415ecd374c9259e5b916c8ae8f29bd13205a1b52decc94c29afc772"
Sep 17 21:27:35.903764 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.903650    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8624785ba415ecd374c9259e5b916c8ae8f29bd13205a1b52decc94c29afc772"} err="failed to get container status \"8624785ba415ecd374c9259e5b916c8ae8f29bd13205a1b52decc94c29afc772\": rpc error: code = NotFound desc = could not find container \"8624785ba415ecd374c9259e5b916c8ae8f29bd13205a1b52decc94c29afc772\": container with ID starting with 8624785ba415ecd374c9259e5b916c8ae8f29bd13205a1b52decc94c29afc772 not found: ID does not exist"
Sep 17 21:27:35.903764 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.903670    2571 scope.go:117] "RemoveContainer" containerID="ed031e90d50910824ce414e1843c8f469be1bbde1a237e6f6e49186e10fb1b51"
Sep 17 21:27:35.903974 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:27:35.903953    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed031e90d50910824ce414e1843c8f469be1bbde1a237e6f6e49186e10fb1b51\": container with ID starting with ed031e90d50910824ce414e1843c8f469be1bbde1a237e6f6e49186e10fb1b51 not found: ID does not exist" containerID="ed031e90d50910824ce414e1843c8f469be1bbde1a237e6f6e49186e10fb1b51"
Sep 17 21:27:35.904063 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.903978    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed031e90d50910824ce414e1843c8f469be1bbde1a237e6f6e49186e10fb1b51"} err="failed to get container status \"ed031e90d50910824ce414e1843c8f469be1bbde1a237e6f6e49186e10fb1b51\": rpc error: code = NotFound desc = could not find container \"ed031e90d50910824ce414e1843c8f469be1bbde1a237e6f6e49186e10fb1b51\": container with ID starting with ed031e90d50910824ce414e1843c8f469be1bbde1a237e6f6e49186e10fb1b51 not found: ID does not exist"
Sep 17 21:27:35.907839 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:35.907821    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9jhp\" (UniqueName: \"kubernetes.io/projected/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kube-api-access-p9jhp\") pod \"isvc-canary-stop-predictor-55656fdf77-nwcrt\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:36.022416 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:36.022376    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:36.298463 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:36.298436    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"]
Sep 17 21:27:36.300422 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:27:36.300393    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a54a1b_012b_46b2_9878_97c2b1973e9c.slice/crio-5c6ef7e2ab9beb482aeea38ea7a78607202b9afc3bb541bee567808b163c758f WatchSource:0}: Error finding container 5c6ef7e2ab9beb482aeea38ea7a78607202b9afc3bb541bee567808b163c758f: Status 404 returned error can't find the container with id 5c6ef7e2ab9beb482aeea38ea7a78607202b9afc3bb541bee567808b163c758f
Sep 17 21:27:36.672157 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:36.672077    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d871933-528e-4880-a65c-7337151e63b3" path="/var/lib/kubelet/pods/6d871933-528e-4880-a65c-7337151e63b3/volumes"
Sep 17 21:27:36.881115 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:36.881080    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" event={"ID":"d5a54a1b-012b-46b2-9878-97c2b1973e9c","Type":"ContainerStarted","Data":"ac450e14bd2d99abcd30d878913f9359236ac4dc4c3e4ecea21e6b5dd06c576d"}
Sep 17 21:27:36.881265 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:36.881148    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" event={"ID":"d5a54a1b-012b-46b2-9878-97c2b1973e9c","Type":"ContainerStarted","Data":"5c6ef7e2ab9beb482aeea38ea7a78607202b9afc3bb541bee567808b163c758f"}
Sep 17 21:27:40.903020 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:40.902988    2571 generic.go:356] "Generic (PLEG): container finished" podID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerID="ac450e14bd2d99abcd30d878913f9359236ac4dc4c3e4ecea21e6b5dd06c576d" exitCode=0
Sep 17 21:27:40.903409 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:40.903060    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" event={"ID":"d5a54a1b-012b-46b2-9878-97c2b1973e9c","Type":"ContainerDied","Data":"ac450e14bd2d99abcd30d878913f9359236ac4dc4c3e4ecea21e6b5dd06c576d"}
Sep 17 21:27:41.913480 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:41.913447    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" event={"ID":"d5a54a1b-012b-46b2-9878-97c2b1973e9c","Type":"ContainerStarted","Data":"b2cea49d0e7367f759cdedba079cfdc15a11e1720eae28edd0c4d2bf4422258c"}
Sep 17 21:27:41.913909 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:41.913511    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" event={"ID":"d5a54a1b-012b-46b2-9878-97c2b1973e9c","Type":"ContainerStarted","Data":"bb1be186e6a24049301072842a291fd0bc211f5cda21421dc19958a43ac198da"}
Sep 17 21:27:41.913909 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:41.913816    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:41.914015 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:41.913966    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:41.915183 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:41.915159    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 21:27:41.936088 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:41.936039    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" podStartSLOduration=3.1761674 podStartE2EDuration="6.936025676s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.759858276s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:27:41.933465245 +0000 UTC m=+2273.437942833" watchObservedRunningTime="2026-09-17 21:27:41.936025676 +0000 UTC m=+2273.440503340"
Sep 17 21:27:42.919026 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:42.918981    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 21:27:47.923636 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:47.923607    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:27:47.924306 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:47.924277    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 21:27:57.924438 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:27:57.924394    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 21:28:07.924814 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:28:07.924761    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 21:28:17.924714 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:28:17.924669    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 21:28:27.924447 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:28:27.924405    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 21:28:37.925211 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:28:37.925119    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 21:28:47.924134 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:28:47.924090    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 21:28:57.673602 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:28:57.673565    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"]
Sep 17 21:28:57.674363 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:28:57.673830    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kserve-container" containerID="cri-o://bb1be186e6a24049301072842a291fd0bc211f5cda21421dc19958a43ac198da" gracePeriod=30
Sep 17 21:28:57.674363 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:28:57.673932    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kube-rbac-proxy" containerID="cri-o://b2cea49d0e7367f759cdedba079cfdc15a11e1720eae28edd0c4d2bf4422258c" gracePeriod=30
Sep 17 21:28:57.674363 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:28:57.674136    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 21:28:57.920243 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:28:57.920199    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 21:28:58.285384 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:28:58.285347    2571 generic.go:356] "Generic (PLEG): container finished" podID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerID="b2cea49d0e7367f759cdedba079cfdc15a11e1720eae28edd0c4d2bf4422258c" exitCode=2
Sep 17 21:28:58.285599 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:28:58.285428    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" event={"ID":"d5a54a1b-012b-46b2-9878-97c2b1973e9c","Type":"ContainerDied","Data":"b2cea49d0e7367f759cdedba079cfdc15a11e1720eae28edd0c4d2bf4422258c"}
Sep 17 21:29:02.919694 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:02.919653    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 21:29:03.126043 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.126017    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:29:03.260203 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.260106    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9jhp\" (UniqueName: \"kubernetes.io/projected/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kube-api-access-p9jhp\") pod \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") "
Sep 17 21:29:03.260203 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.260164    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5a54a1b-012b-46b2-9878-97c2b1973e9c-proxy-tls\") pod \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") "
Sep 17 21:29:03.260446 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.260211    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kube-rbac-proxy-sar-config\") pod \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") "
Sep 17 21:29:03.260446 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.260226    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a54a1b-012b-46b2-9878-97c2b1973e9c-cabundle-cert\") pod \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") "
Sep 17 21:29:03.260446 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.260268    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kserve-provision-location\") pod \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\" (UID: \"d5a54a1b-012b-46b2-9878-97c2b1973e9c\") "
Sep 17 21:29:03.260691 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.260666    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5a54a1b-012b-46b2-9878-97c2b1973e9c" (UID: "d5a54a1b-012b-46b2-9878-97c2b1973e9c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:29:03.260691 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.260679    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5a54a1b-012b-46b2-9878-97c2b1973e9c" (UID: "d5a54a1b-012b-46b2-9878-97c2b1973e9c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:29:03.260782 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.260662    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a54a1b-012b-46b2-9878-97c2b1973e9c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5a54a1b-012b-46b2-9878-97c2b1973e9c" (UID: "d5a54a1b-012b-46b2-9878-97c2b1973e9c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:29:03.262598 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.262572    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kube-api-access-p9jhp" (OuterVolumeSpecName: "kube-api-access-p9jhp") pod "d5a54a1b-012b-46b2-9878-97c2b1973e9c" (UID: "d5a54a1b-012b-46b2-9878-97c2b1973e9c"). InnerVolumeSpecName "kube-api-access-p9jhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:29:03.262791 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.262636    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a54a1b-012b-46b2-9878-97c2b1973e9c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5a54a1b-012b-46b2-9878-97c2b1973e9c" (UID: "d5a54a1b-012b-46b2-9878-97c2b1973e9c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:29:03.312042 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.311995    2571 generic.go:356] "Generic (PLEG): container finished" podID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerID="bb1be186e6a24049301072842a291fd0bc211f5cda21421dc19958a43ac198da" exitCode=139
Sep 17 21:29:03.312225 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.312054    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"
Sep 17 21:29:03.312225 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.312086    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" event={"ID":"d5a54a1b-012b-46b2-9878-97c2b1973e9c","Type":"ContainerDied","Data":"bb1be186e6a24049301072842a291fd0bc211f5cda21421dc19958a43ac198da"}
Sep 17 21:29:03.312225 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.312147    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt" event={"ID":"d5a54a1b-012b-46b2-9878-97c2b1973e9c","Type":"ContainerDied","Data":"5c6ef7e2ab9beb482aeea38ea7a78607202b9afc3bb541bee567808b163c758f"}
Sep 17 21:29:03.312225 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.312174    2571 scope.go:117] "RemoveContainer" containerID="b2cea49d0e7367f759cdedba079cfdc15a11e1720eae28edd0c4d2bf4422258c"
Sep 17 21:29:03.321991 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.321968    2571 scope.go:117] "RemoveContainer" containerID="bb1be186e6a24049301072842a291fd0bc211f5cda21421dc19958a43ac198da"
Sep 17 21:29:03.330185 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.330164    2571 scope.go:117] "RemoveContainer" containerID="ac450e14bd2d99abcd30d878913f9359236ac4dc4c3e4ecea21e6b5dd06c576d"
Sep 17 21:29:03.335877 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.335849    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"]
Sep 17 21:29:03.338987 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.338966    2571 scope.go:117] "RemoveContainer" containerID="b2cea49d0e7367f759cdedba079cfdc15a11e1720eae28edd0c4d2bf4422258c"
Sep 17 21:29:03.339301 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:29:03.339283    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2cea49d0e7367f759cdedba079cfdc15a11e1720eae28edd0c4d2bf4422258c\": container with ID starting with b2cea49d0e7367f759cdedba079cfdc15a11e1720eae28edd0c4d2bf4422258c not found: ID does not exist" containerID="b2cea49d0e7367f759cdedba079cfdc15a11e1720eae28edd0c4d2bf4422258c"
Sep 17 21:29:03.339379 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.339309    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2cea49d0e7367f759cdedba079cfdc15a11e1720eae28edd0c4d2bf4422258c"} err="failed to get container status \"b2cea49d0e7367f759cdedba079cfdc15a11e1720eae28edd0c4d2bf4422258c\": rpc error: code = NotFound desc = could not find container \"b2cea49d0e7367f759cdedba079cfdc15a11e1720eae28edd0c4d2bf4422258c\": container with ID starting with b2cea49d0e7367f759cdedba079cfdc15a11e1720eae28edd0c4d2bf4422258c not found: ID does not exist"
Sep 17 21:29:03.339379 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.339337    2571 scope.go:117] "RemoveContainer" containerID="bb1be186e6a24049301072842a291fd0bc211f5cda21421dc19958a43ac198da"
Sep 17 21:29:03.339648 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:29:03.339627    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb1be186e6a24049301072842a291fd0bc211f5cda21421dc19958a43ac198da\": container with ID starting with bb1be186e6a24049301072842a291fd0bc211f5cda21421dc19958a43ac198da not found: ID does not exist" containerID="bb1be186e6a24049301072842a291fd0bc211f5cda21421dc19958a43ac198da"
Sep 17 21:29:03.339696 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.339655    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1be186e6a24049301072842a291fd0bc211f5cda21421dc19958a43ac198da"} err="failed to get container status \"bb1be186e6a24049301072842a291fd0bc211f5cda21421dc19958a43ac198da\": rpc error: code = NotFound desc = could not find container \"bb1be186e6a24049301072842a291fd0bc211f5cda21421dc19958a43ac198da\": container with ID starting with bb1be186e6a24049301072842a291fd0bc211f5cda21421dc19958a43ac198da not found: ID does not exist"
Sep 17 21:29:03.339696 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.339672    2571 scope.go:117] "RemoveContainer" containerID="ac450e14bd2d99abcd30d878913f9359236ac4dc4c3e4ecea21e6b5dd06c576d"
Sep 17 21:29:03.339949 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:29:03.339931    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac450e14bd2d99abcd30d878913f9359236ac4dc4c3e4ecea21e6b5dd06c576d\": container with ID starting with ac450e14bd2d99abcd30d878913f9359236ac4dc4c3e4ecea21e6b5dd06c576d not found: ID does not exist" containerID="ac450e14bd2d99abcd30d878913f9359236ac4dc4c3e4ecea21e6b5dd06c576d"
Sep 17 21:29:03.339993 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.339954    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac450e14bd2d99abcd30d878913f9359236ac4dc4c3e4ecea21e6b5dd06c576d"} err="failed to get container status \"ac450e14bd2d99abcd30d878913f9359236ac4dc4c3e4ecea21e6b5dd06c576d\": rpc error: code = NotFound desc = could not find container \"ac450e14bd2d99abcd30d878913f9359236ac4dc4c3e4ecea21e6b5dd06c576d\": container with ID starting with ac450e14bd2d99abcd30d878913f9359236ac4dc4c3e4ecea21e6b5dd06c576d not found: ID does not exist"
Sep 17 21:29:03.340064 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.340046    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-nwcrt"]
Sep 17 21:29:03.361526 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.361495    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:29:03.361526 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.361524    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a54a1b-012b-46b2-9878-97c2b1973e9c-cabundle-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:29:03.361738 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.361536    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kserve-provision-location\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:29:03.361738 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.361563    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9jhp\" (UniqueName: \"kubernetes.io/projected/d5a54a1b-012b-46b2-9878-97c2b1973e9c-kube-api-access-p9jhp\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:29:03.361738 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:03.361573    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5a54a1b-012b-46b2-9878-97c2b1973e9c-proxy-tls\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:29:04.671057 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:29:04.671021    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" path="/var/lib/kubelet/pods/d5a54a1b-012b-46b2-9878-97c2b1973e9c/volumes"
Sep 17 21:36:36.201302 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.201217    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"]
Sep 17 21:36:36.201810 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.201620    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="storage-initializer"
Sep 17 21:36:36.201810 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.201632    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="storage-initializer"
Sep 17 21:36:36.201810 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.201650    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kserve-container"
Sep 17 21:36:36.201810 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.201656    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kserve-container"
Sep 17 21:36:36.201810 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.201666    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kube-rbac-proxy"
Sep 17 21:36:36.201810 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.201672    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kube-rbac-proxy"
Sep 17 21:36:36.201810 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.201733    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kserve-container"
Sep 17 21:36:36.201810 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.201745    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a54a1b-012b-46b2-9878-97c2b1973e9c" containerName="kube-rbac-proxy"
Sep 17 21:36:36.205125 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.205104    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"
Sep 17 21:36:36.209383 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.209356    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 21:36:36.209868 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.209852    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 17 21:36:36.210248 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.210228    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:36:36.210358 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.210233    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:36:36.210358 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.210235    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 17 21:36:36.210475 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.210370    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:36:36.217674 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.217652    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"]
Sep 17 21:36:36.396837 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.396791    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94806b38-2be5-416a-af81-6dfd50a4459a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc\" (UID: \"94806b38-2be5-416a-af81-6dfd50a4459a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"
Sep 17 21:36:36.396837 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.396841    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pddd\" (UniqueName: \"kubernetes.io/projected/94806b38-2be5-416a-af81-6dfd50a4459a-kube-api-access-8pddd\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc\" (UID: \"94806b38-2be5-416a-af81-6dfd50a4459a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"
Sep 17 21:36:36.397088 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.396936    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94806b38-2be5-416a-af81-6dfd50a4459a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc\" (UID: \"94806b38-2be5-416a-af81-6dfd50a4459a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"
Sep 17 21:36:36.397088 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.396955    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94806b38-2be5-416a-af81-6dfd50a4459a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc\" (UID: \"94806b38-2be5-416a-af81-6dfd50a4459a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"
Sep 17 21:36:36.498194 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.498102    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94806b38-2be5-416a-af81-6dfd50a4459a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc\" (UID: \"94806b38-2be5-416a-af81-6dfd50a4459a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"
Sep 17 21:36:36.498194 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.498135    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94806b38-2be5-416a-af81-6dfd50a4459a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc\" (UID: \"94806b38-2be5-416a-af81-6dfd50a4459a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"
Sep 17 21:36:36.498194 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.498185    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94806b38-2be5-416a-af81-6dfd50a4459a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc\" (UID: \"94806b38-2be5-416a-af81-6dfd50a4459a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"
Sep 17 21:36:36.498520 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.498212    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pddd\" (UniqueName: \"kubernetes.io/projected/94806b38-2be5-416a-af81-6dfd50a4459a-kube-api-access-8pddd\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc\" (UID: \"94806b38-2be5-416a-af81-6dfd50a4459a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"
Sep 17 21:36:36.498656 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.498616    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94806b38-2be5-416a-af81-6dfd50a4459a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc\" (UID: \"94806b38-2be5-416a-af81-6dfd50a4459a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"
Sep 17 21:36:36.498949 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.498930    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94806b38-2be5-416a-af81-6dfd50a4459a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc\" (UID: \"94806b38-2be5-416a-af81-6dfd50a4459a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"
Sep 17 21:36:36.500984 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.500966    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94806b38-2be5-416a-af81-6dfd50a4459a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc\" (UID: \"94806b38-2be5-416a-af81-6dfd50a4459a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"
Sep 17 21:36:36.508223 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.508199    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pddd\" (UniqueName: \"kubernetes.io/projected/94806b38-2be5-416a-af81-6dfd50a4459a-kube-api-access-8pddd\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc\" (UID: \"94806b38-2be5-416a-af81-6dfd50a4459a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"
Sep 17 21:36:36.515929 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.515905    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"
Sep 17 21:36:36.799832 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:36.799756    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"]
Sep 17 21:36:36.803395 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:36:36.803366    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94806b38_2be5_416a_af81_6dfd50a4459a.slice/crio-6e0dd1faca443d68dc3cf98d8c676f95de1096e50c1b603c84265b9f8d2091d9 WatchSource:0}: Error finding container 6e0dd1faca443d68dc3cf98d8c676f95de1096e50c1b603c84265b9f8d2091d9: Status 404 returned error can't find the container with id 6e0dd1faca443d68dc3cf98d8c676f95de1096e50c1b603c84265b9f8d2091d9
Sep 17 21:36:37.505845 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:37.505803    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc" event={"ID":"94806b38-2be5-416a-af81-6dfd50a4459a","Type":"ContainerStarted","Data":"5a60ce74f354e8b7127da0d02645d5fd8fb24fca647dc03a236d1e9dcb72de93"}
Sep 17 21:36:37.506327 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:37.505907    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc" event={"ID":"94806b38-2be5-416a-af81-6dfd50a4459a","Type":"ContainerStarted","Data":"6e0dd1faca443d68dc3cf98d8c676f95de1096e50c1b603c84265b9f8d2091d9"}
Sep 17 21:36:40.522190 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:40.522156    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc_94806b38-2be5-416a-af81-6dfd50a4459a/storage-initializer/0.log"
Sep 17 21:36:40.522589 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:40.522195    2571 generic.go:356] "Generic (PLEG): container finished" podID="94806b38-2be5-416a-af81-6dfd50a4459a" containerID="5a60ce74f354e8b7127da0d02645d5fd8fb24fca647dc03a236d1e9dcb72de93" exitCode=1
Sep 17 21:36:40.522589 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:40.522278    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc" event={"ID":"94806b38-2be5-416a-af81-6dfd50a4459a","Type":"ContainerDied","Data":"5a60ce74f354e8b7127da0d02645d5fd8fb24fca647dc03a236d1e9dcb72de93"}
Sep 17 21:36:41.529181 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:41.529153    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc_94806b38-2be5-416a-af81-6dfd50a4459a/storage-initializer/0.log"
Sep 17 21:36:41.529625 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:41.529248    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc" event={"ID":"94806b38-2be5-416a-af81-6dfd50a4459a","Type":"ContainerStarted","Data":"52bf9a3c8de4bbce7e47afa19455b7fcef3c7a71fba6a8e7c74e0a2556e90609"}
Sep 17 21:36:43.540945 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:43.540917    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc_94806b38-2be5-416a-af81-6dfd50a4459a/storage-initializer/1.log"
Sep 17 21:36:43.541401 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:43.541382    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc_94806b38-2be5-416a-af81-6dfd50a4459a/storage-initializer/0.log"
Sep 17 21:36:43.541471 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:43.541427    2571 generic.go:356] "Generic (PLEG): container finished" podID="94806b38-2be5-416a-af81-6dfd50a4459a" containerID="52bf9a3c8de4bbce7e47afa19455b7fcef3c7a71fba6a8e7c74e0a2556e90609" exitCode=1
Sep 17 21:36:43.541519 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:43.541502    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc" event={"ID":"94806b38-2be5-416a-af81-6dfd50a4459a","Type":"ContainerDied","Data":"52bf9a3c8de4bbce7e47afa19455b7fcef3c7a71fba6a8e7c74e0a2556e90609"}
Sep 17 21:36:43.541590 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:43.541583    2571 scope.go:117] "RemoveContainer" containerID="5a60ce74f354e8b7127da0d02645d5fd8fb24fca647dc03a236d1e9dcb72de93"
Sep 17 21:36:43.541987 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:43.541968    2571 scope.go:117] "RemoveContainer" containerID="5a60ce74f354e8b7127da0d02645d5fd8fb24fca647dc03a236d1e9dcb72de93"
Sep 17 21:36:43.552821 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:36:43.552802    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc_kserve-ci-e2e-test(94806b38-2be5-416a-af81-6dfd50a4459a)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc" podUID="94806b38-2be5-416a-af81-6dfd50a4459a"
Sep 17 21:36:43.552937 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:36:43.552904    2571 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc_kserve-ci-e2e-test_94806b38-2be5-416a-af81-6dfd50a4459a_0 in pod sandbox 6e0dd1faca443d68dc3cf98d8c676f95de1096e50c1b603c84265b9f8d2091d9 from index: no such id: '5a60ce74f354e8b7127da0d02645d5fd8fb24fca647dc03a236d1e9dcb72de93'" containerID="5a60ce74f354e8b7127da0d02645d5fd8fb24fca647dc03a236d1e9dcb72de93"
Sep 17 21:36:43.553004 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:43.552952    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a60ce74f354e8b7127da0d02645d5fd8fb24fca647dc03a236d1e9dcb72de93"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc_kserve-ci-e2e-test_94806b38-2be5-416a-af81-6dfd50a4459a_0 in pod sandbox 6e0dd1faca443d68dc3cf98d8c676f95de1096e50c1b603c84265b9f8d2091d9 from index: no such id: '5a60ce74f354e8b7127da0d02645d5fd8fb24fca647dc03a236d1e9dcb72de93'"
Sep 17 21:36:44.548663 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:44.548634    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc_94806b38-2be5-416a-af81-6dfd50a4459a/storage-initializer/1.log"
Sep 17 21:36:46.186746 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.186708    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"]
Sep 17 21:36:46.319925 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.319901    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc_94806b38-2be5-416a-af81-6dfd50a4459a/storage-initializer/1.log"
Sep 17 21:36:46.323421 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.323400    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"
Sep 17 21:36:46.386397 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.386364    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94806b38-2be5-416a-af81-6dfd50a4459a-kserve-provision-location\") pod \"94806b38-2be5-416a-af81-6dfd50a4459a\" (UID: \"94806b38-2be5-416a-af81-6dfd50a4459a\") "
Sep 17 21:36:46.386590 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.386414    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94806b38-2be5-416a-af81-6dfd50a4459a-proxy-tls\") pod \"94806b38-2be5-416a-af81-6dfd50a4459a\" (UID: \"94806b38-2be5-416a-af81-6dfd50a4459a\") "
Sep 17 21:36:46.386590 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.386453    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pddd\" (UniqueName: \"kubernetes.io/projected/94806b38-2be5-416a-af81-6dfd50a4459a-kube-api-access-8pddd\") pod \"94806b38-2be5-416a-af81-6dfd50a4459a\" (UID: \"94806b38-2be5-416a-af81-6dfd50a4459a\") "
Sep 17 21:36:46.386590 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.386502    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94806b38-2be5-416a-af81-6dfd50a4459a-kube-rbac-proxy-sar-config\") pod \"94806b38-2be5-416a-af81-6dfd50a4459a\" (UID: \"94806b38-2be5-416a-af81-6dfd50a4459a\") "
Sep 17 21:36:46.386781 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.386695    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94806b38-2be5-416a-af81-6dfd50a4459a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "94806b38-2be5-416a-af81-6dfd50a4459a" (UID: "94806b38-2be5-416a-af81-6dfd50a4459a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:46.386842 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.386805    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94806b38-2be5-416a-af81-6dfd50a4459a-kserve-provision-location\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:36:46.386897 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.386875    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94806b38-2be5-416a-af81-6dfd50a4459a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "94806b38-2be5-416a-af81-6dfd50a4459a" (UID: "94806b38-2be5-416a-af81-6dfd50a4459a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:36:46.388673 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.388653    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94806b38-2be5-416a-af81-6dfd50a4459a-kube-api-access-8pddd" (OuterVolumeSpecName: "kube-api-access-8pddd") pod "94806b38-2be5-416a-af81-6dfd50a4459a" (UID: "94806b38-2be5-416a-af81-6dfd50a4459a"). InnerVolumeSpecName "kube-api-access-8pddd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:36:46.388790 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.388771    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94806b38-2be5-416a-af81-6dfd50a4459a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "94806b38-2be5-416a-af81-6dfd50a4459a" (UID: "94806b38-2be5-416a-af81-6dfd50a4459a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:36:46.488102 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.488016    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94806b38-2be5-416a-af81-6dfd50a4459a-proxy-tls\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:36:46.488102 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.488045    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pddd\" (UniqueName: \"kubernetes.io/projected/94806b38-2be5-416a-af81-6dfd50a4459a-kube-api-access-8pddd\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:36:46.488102 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.488055    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94806b38-2be5-416a-af81-6dfd50a4459a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:36:46.560810 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.560783    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc_94806b38-2be5-416a-af81-6dfd50a4459a/storage-initializer/1.log"
Sep 17 21:36:46.560964 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.560874    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"
Sep 17 21:36:46.560964 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.560871    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc" event={"ID":"94806b38-2be5-416a-af81-6dfd50a4459a","Type":"ContainerDied","Data":"6e0dd1faca443d68dc3cf98d8c676f95de1096e50c1b603c84265b9f8d2091d9"}
Sep 17 21:36:46.561037 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.561003    2571 scope.go:117] "RemoveContainer" containerID="52bf9a3c8de4bbce7e47afa19455b7fcef3c7a71fba6a8e7c74e0a2556e90609"
Sep 17 21:36:46.597853 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.597820    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"]
Sep 17 21:36:46.602838 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.602810    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-648df7b6d5-wnmsc"]
Sep 17 21:36:46.670278 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:46.670249    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94806b38-2be5-416a-af81-6dfd50a4459a" path="/var/lib/kubelet/pods/94806b38-2be5-416a-af81-6dfd50a4459a/volumes"
Sep 17 21:36:47.354815 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.354782    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"]
Sep 17 21:36:47.355190 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.355141    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94806b38-2be5-416a-af81-6dfd50a4459a" containerName="storage-initializer"
Sep 17 21:36:47.355190 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.355152    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="94806b38-2be5-416a-af81-6dfd50a4459a" containerName="storage-initializer"
Sep 17 21:36:47.355265 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.355225    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="94806b38-2be5-416a-af81-6dfd50a4459a" containerName="storage-initializer"
Sep 17 21:36:47.355265 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.355236    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="94806b38-2be5-416a-af81-6dfd50a4459a" containerName="storage-initializer"
Sep 17 21:36:47.355325 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.355288    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94806b38-2be5-416a-af81-6dfd50a4459a" containerName="storage-initializer"
Sep 17 21:36:47.355325 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.355295    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="94806b38-2be5-416a-af81-6dfd50a4459a" containerName="storage-initializer"
Sep 17 21:36:47.358424 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.358407    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:47.361563 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.361526    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 17 21:36:47.361773 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.361754    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:36:47.361822 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.361810    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:36:47.362049 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.362036    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 21:36:47.362621 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.362604    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:36:47.362687 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.362605    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 17 21:36:47.362801 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.362789    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:36:47.372893 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.372873    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"]
Sep 17 21:36:47.394444 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.394410    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:47.394444 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.394443    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4vtk\" (UniqueName: \"kubernetes.io/projected/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kube-api-access-t4vtk\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:47.394671 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.394526    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:47.394671 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.394608    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:47.394671 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.394645    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:47.495103 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.495050    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:47.495103 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.495104    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vtk\" (UniqueName: \"kubernetes.io/projected/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kube-api-access-t4vtk\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:47.495374 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.495172    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:47.495374 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.495200    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:47.495374 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.495233    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:47.495374 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:36:47.495343    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 17 21:36:47.495632 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:36:47.495435    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-proxy-tls podName:d009aa3b-aa26-41b3-b183-e3b615fa9b8a nodeName:}" failed. No retries permitted until 2026-09-17 21:36:47.995414321 +0000 UTC m=+2819.499891895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" (UID: "d009aa3b-aa26-41b3-b183-e3b615fa9b8a") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 17 21:36:47.495632 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.495575    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:47.495836 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.495814    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:47.495987 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.495969    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:47.503991 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.503967    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4vtk\" (UniqueName: \"kubernetes.io/projected/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kube-api-access-t4vtk\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:47.999839 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:47.999805    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:48.002456 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:48.002431    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:48.269791 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:48.269696    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:48.544746 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:48.544671    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"]
Sep 17 21:36:48.547742 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:36:48.547711    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd009aa3b_aa26_41b3_b183_e3b615fa9b8a.slice/crio-ca6db159114ccf3f47ad3e5d6ea59135c366e28fa18888dfe962987dad2fa78f WatchSource:0}: Error finding container ca6db159114ccf3f47ad3e5d6ea59135c366e28fa18888dfe962987dad2fa78f: Status 404 returned error can't find the container with id ca6db159114ccf3f47ad3e5d6ea59135c366e28fa18888dfe962987dad2fa78f
Sep 17 21:36:48.573456 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:48.573429    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" event={"ID":"d009aa3b-aa26-41b3-b183-e3b615fa9b8a","Type":"ContainerStarted","Data":"ca6db159114ccf3f47ad3e5d6ea59135c366e28fa18888dfe962987dad2fa78f"}
Sep 17 21:36:49.580475 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:49.580442    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" event={"ID":"d009aa3b-aa26-41b3-b183-e3b615fa9b8a","Type":"ContainerStarted","Data":"7b8aa7507f5a36e1ee5278051f94185ef642404ecf9c0d7dd650ed681f0d946b"}
Sep 17 21:36:50.587378 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:50.587345    2571 generic.go:356] "Generic (PLEG): container finished" podID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerID="7b8aa7507f5a36e1ee5278051f94185ef642404ecf9c0d7dd650ed681f0d946b" exitCode=0
Sep 17 21:36:50.587829 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:50.587433    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" event={"ID":"d009aa3b-aa26-41b3-b183-e3b615fa9b8a","Type":"ContainerDied","Data":"7b8aa7507f5a36e1ee5278051f94185ef642404ecf9c0d7dd650ed681f0d946b"}
Sep 17 21:36:51.595401 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:51.595367    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" event={"ID":"d009aa3b-aa26-41b3-b183-e3b615fa9b8a","Type":"ContainerStarted","Data":"c26905f26b66f00dd53699c7e4143d892dd8f8546d14056e6f0298c9fec98774"}
Sep 17 21:36:51.595845 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:51.595431    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:51.595845 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:51.595463    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" event={"ID":"d009aa3b-aa26-41b3-b183-e3b615fa9b8a","Type":"ContainerStarted","Data":"cbdaf75f3e5e85ba31f71311bc6f42418df1b846b760b9439e45dc9e1e926476"}
Sep 17 21:36:51.595845 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:51.595612    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:51.596925 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:51.596901    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 21:36:51.615107 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:51.615062    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" podStartSLOduration=3.572733569 podStartE2EDuration="4.615049487s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.042315918s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:36:51.613814951 +0000 UTC m=+2823.118292549" watchObservedRunningTime="2026-09-17 21:36:51.615049487 +0000 UTC m=+2823.119527082"
Sep 17 21:36:52.600917 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:52.600879    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 21:36:57.605180 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:57.605149    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:36:57.605614 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:36:57.605592    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 21:37:07.605814 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:37:07.605766    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 21:37:17.606339 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:37:17.606291    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 21:37:27.605976 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:37:27.605933    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 21:37:37.605665 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:37:37.605580    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 21:37:47.606113 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:37:47.606069    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 21:37:57.606147 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:37:57.606109    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 21:38:07.606003 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:07.605959    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 21:38:17.606722 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:17.606689    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:38:27.341909 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:27.341874    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"]
Sep 17 21:38:27.342420 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:27.342159    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kserve-container" containerID="cri-o://cbdaf75f3e5e85ba31f71311bc6f42418df1b846b760b9439e45dc9e1e926476" gracePeriod=30
Sep 17 21:38:27.342420 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:27.342202    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kube-rbac-proxy" containerID="cri-o://c26905f26b66f00dd53699c7e4143d892dd8f8546d14056e6f0298c9fec98774" gracePeriod=30
Sep 17 21:38:27.601520 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:27.601417    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Sep 17 21:38:27.605758 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:27.605730    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 21:38:28.071397 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:28.071366    2571 generic.go:356] "Generic (PLEG): container finished" podID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerID="c26905f26b66f00dd53699c7e4143d892dd8f8546d14056e6f0298c9fec98774" exitCode=2
Sep 17 21:38:28.071612 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:28.071445    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" event={"ID":"d009aa3b-aa26-41b3-b183-e3b615fa9b8a","Type":"ContainerDied","Data":"c26905f26b66f00dd53699c7e4143d892dd8f8546d14056e6f0298c9fec98774"}
Sep 17 21:38:32.396247 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:32.396215    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:38:32.411690 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:32.411666    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kserve-provision-location\") pod \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") "
Sep 17 21:38:32.411816 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:32.411723    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-proxy-tls\") pod \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") "
Sep 17 21:38:32.411816 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:32.411769    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4vtk\" (UniqueName: \"kubernetes.io/projected/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kube-api-access-t4vtk\") pod \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") "
Sep 17 21:38:32.411816 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:32.411797    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kube-rbac-proxy-sar-config\") pod \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") "
Sep 17 21:38:32.411978 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:32.411851    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-cabundle-cert\") pod \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\" (UID: \"d009aa3b-aa26-41b3-b183-e3b615fa9b8a\") "
Sep 17 21:38:32.411978 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:32.411955    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d009aa3b-aa26-41b3-b183-e3b615fa9b8a" (UID: "d009aa3b-aa26-41b3-b183-e3b615fa9b8a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:38:32.412279 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:32.412255    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kserve-provision-location\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:38:32.412381 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:32.412276    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d009aa3b-aa26-41b3-b183-e3b615fa9b8a" (UID: "d009aa3b-aa26-41b3-b183-e3b615fa9b8a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:38:32.412381 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:32.412339    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d009aa3b-aa26-41b3-b183-e3b615fa9b8a" (UID: "d009aa3b-aa26-41b3-b183-e3b615fa9b8a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:38:32.414141 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:32.414119    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d009aa3b-aa26-41b3-b183-e3b615fa9b8a" (UID: "d009aa3b-aa26-41b3-b183-e3b615fa9b8a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:38:32.414480 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:32.414463    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kube-api-access-t4vtk" (OuterVolumeSpecName: "kube-api-access-t4vtk") pod "d009aa3b-aa26-41b3-b183-e3b615fa9b8a" (UID: "d009aa3b-aa26-41b3-b183-e3b615fa9b8a"). InnerVolumeSpecName "kube-api-access-t4vtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:38:32.512707 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:32.512621    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-cabundle-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:38:32.512707 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:32.512650    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-proxy-tls\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:38:32.512707 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:32.512660    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4vtk\" (UniqueName: \"kubernetes.io/projected/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kube-api-access-t4vtk\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:38:32.512707 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:32.512669    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d009aa3b-aa26-41b3-b183-e3b615fa9b8a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:38:33.096351 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:33.096319    2571 generic.go:356] "Generic (PLEG): container finished" podID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerID="cbdaf75f3e5e85ba31f71311bc6f42418df1b846b760b9439e45dc9e1e926476" exitCode=0
Sep 17 21:38:33.096569 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:33.096369    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"
Sep 17 21:38:33.096569 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:33.096406    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" event={"ID":"d009aa3b-aa26-41b3-b183-e3b615fa9b8a","Type":"ContainerDied","Data":"cbdaf75f3e5e85ba31f71311bc6f42418df1b846b760b9439e45dc9e1e926476"}
Sep 17 21:38:33.096569 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:33.096479    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn" event={"ID":"d009aa3b-aa26-41b3-b183-e3b615fa9b8a","Type":"ContainerDied","Data":"ca6db159114ccf3f47ad3e5d6ea59135c366e28fa18888dfe962987dad2fa78f"}
Sep 17 21:38:33.096569 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:33.096517    2571 scope.go:117] "RemoveContainer" containerID="c26905f26b66f00dd53699c7e4143d892dd8f8546d14056e6f0298c9fec98774"
Sep 17 21:38:33.106377 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:33.106359    2571 scope.go:117] "RemoveContainer" containerID="cbdaf75f3e5e85ba31f71311bc6f42418df1b846b760b9439e45dc9e1e926476"
Sep 17 21:38:33.115179 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:33.115134    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"]
Sep 17 21:38:33.116703 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:33.116676    2571 scope.go:117] "RemoveContainer" containerID="7b8aa7507f5a36e1ee5278051f94185ef642404ecf9c0d7dd650ed681f0d946b"
Sep 17 21:38:33.120586 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:33.120565    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-hjtwn"]
Sep 17 21:38:33.125976 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:33.125962    2571 scope.go:117] "RemoveContainer" containerID="c26905f26b66f00dd53699c7e4143d892dd8f8546d14056e6f0298c9fec98774"
Sep 17 21:38:33.126285 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:38:33.126267    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c26905f26b66f00dd53699c7e4143d892dd8f8546d14056e6f0298c9fec98774\": container with ID starting with c26905f26b66f00dd53699c7e4143d892dd8f8546d14056e6f0298c9fec98774 not found: ID does not exist" containerID="c26905f26b66f00dd53699c7e4143d892dd8f8546d14056e6f0298c9fec98774"
Sep 17 21:38:33.126326 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:33.126295    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c26905f26b66f00dd53699c7e4143d892dd8f8546d14056e6f0298c9fec98774"} err="failed to get container status \"c26905f26b66f00dd53699c7e4143d892dd8f8546d14056e6f0298c9fec98774\": rpc error: code = NotFound desc = could not find container \"c26905f26b66f00dd53699c7e4143d892dd8f8546d14056e6f0298c9fec98774\": container with ID starting with c26905f26b66f00dd53699c7e4143d892dd8f8546d14056e6f0298c9fec98774 not found: ID does not exist"
Sep 17 21:38:33.126326 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:33.126313    2571 scope.go:117] "RemoveContainer" containerID="cbdaf75f3e5e85ba31f71311bc6f42418df1b846b760b9439e45dc9e1e926476"
Sep 17 21:38:33.127282 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:38:33.127256    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbdaf75f3e5e85ba31f71311bc6f42418df1b846b760b9439e45dc9e1e926476\": container with ID starting with cbdaf75f3e5e85ba31f71311bc6f42418df1b846b760b9439e45dc9e1e926476 not found: ID does not exist" containerID="cbdaf75f3e5e85ba31f71311bc6f42418df1b846b760b9439e45dc9e1e926476"
Sep 17 21:38:33.127466 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:33.127426    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbdaf75f3e5e85ba31f71311bc6f42418df1b846b760b9439e45dc9e1e926476"} err="failed to get container status \"cbdaf75f3e5e85ba31f71311bc6f42418df1b846b760b9439e45dc9e1e926476\": rpc error: code = NotFound desc = could not find container \"cbdaf75f3e5e85ba31f71311bc6f42418df1b846b760b9439e45dc9e1e926476\": container with ID starting with cbdaf75f3e5e85ba31f71311bc6f42418df1b846b760b9439e45dc9e1e926476 not found: ID does not exist"
Sep 17 21:38:33.127575 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:33.127465    2571 scope.go:117] "RemoveContainer" containerID="7b8aa7507f5a36e1ee5278051f94185ef642404ecf9c0d7dd650ed681f0d946b"
Sep 17 21:38:33.127938 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:38:33.127906    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b8aa7507f5a36e1ee5278051f94185ef642404ecf9c0d7dd650ed681f0d946b\": container with ID starting with 7b8aa7507f5a36e1ee5278051f94185ef642404ecf9c0d7dd650ed681f0d946b not found: ID does not exist" containerID="7b8aa7507f5a36e1ee5278051f94185ef642404ecf9c0d7dd650ed681f0d946b"
Sep 17 21:38:33.128574 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:33.128081    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b8aa7507f5a36e1ee5278051f94185ef642404ecf9c0d7dd650ed681f0d946b"} err="failed to get container status \"7b8aa7507f5a36e1ee5278051f94185ef642404ecf9c0d7dd650ed681f0d946b\": rpc error: code = NotFound desc = could not find container \"7b8aa7507f5a36e1ee5278051f94185ef642404ecf9c0d7dd650ed681f0d946b\": container with ID starting with 7b8aa7507f5a36e1ee5278051f94185ef642404ecf9c0d7dd650ed681f0d946b not found: ID does not exist"
Sep 17 21:38:34.670717 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:34.670683    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" path="/var/lib/kubelet/pods/d009aa3b-aa26-41b3-b183-e3b615fa9b8a/volumes"
Sep 17 21:38:39.564665 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.564628    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"]
Sep 17 21:38:39.565180 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.565158    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kube-rbac-proxy"
Sep 17 21:38:39.565264 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.565183    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kube-rbac-proxy"
Sep 17 21:38:39.565264 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.565221    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kserve-container"
Sep 17 21:38:39.565264 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.565231    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kserve-container"
Sep 17 21:38:39.565264 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.565244    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="storage-initializer"
Sep 17 21:38:39.565264 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.565253    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="storage-initializer"
Sep 17 21:38:39.565513 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.565352    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kserve-container"
Sep 17 21:38:39.565513 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.565365    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d009aa3b-aa26-41b3-b183-e3b615fa9b8a" containerName="kube-rbac-proxy"
Sep 17 21:38:39.570282 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.570260    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:39.573118 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.573088    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:38:39.573337 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.573315    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 17 21:38:39.573425 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.573349    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 17 21:38:39.574118 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.574090    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:38:39.574118 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.574115    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:38:39.574294 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.574100    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 21:38:39.574294 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.574134    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:38:39.581218 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.581194    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"]
Sep 17 21:38:39.675203 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.675166    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:39.675203 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.675204    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74qkb\" (UniqueName: \"kubernetes.io/projected/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kube-api-access-74qkb\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:39.675406 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.675284    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:39.675406 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.675306    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:39.675406 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.675352    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:39.776602 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.776543    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:39.776774 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.776611    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:39.776774 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.776632    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:39.776774 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.776730    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:39.776774 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:38:39.776740    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 17 21:38:39.776774 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.776764    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74qkb\" (UniqueName: \"kubernetes.io/projected/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kube-api-access-74qkb\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:39.776986 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:38:39.776808    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-proxy-tls podName:d3e9829c-4fae-4c39-a7c9-72b3d7b612e2 nodeName:}" failed. No retries permitted until 2026-09-17 21:38:40.276784853 +0000 UTC m=+2931.781262427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" (UID: "d3e9829c-4fae-4c39-a7c9-72b3d7b612e2") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 17 21:38:39.777108 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.777062    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:39.777387 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.777365    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:39.777432 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.777414    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:39.796762 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:39.796729    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74qkb\" (UniqueName: \"kubernetes.io/projected/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kube-api-access-74qkb\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:40.280813 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:40.280767    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:40.283379 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:40.283356    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:40.483121 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:40.483087    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:40.758681 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:40.758655    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"]
Sep 17 21:38:40.761299 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:38:40.761273    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e9829c_4fae_4c39_a7c9_72b3d7b612e2.slice/crio-7cb3741ffe575a5b7fa0f61284ed7e0bacbdd2b76912b0e687f81a3999f27173 WatchSource:0}: Error finding container 7cb3741ffe575a5b7fa0f61284ed7e0bacbdd2b76912b0e687f81a3999f27173: Status 404 returned error can't find the container with id 7cb3741ffe575a5b7fa0f61284ed7e0bacbdd2b76912b0e687f81a3999f27173
Sep 17 21:38:41.140933 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:41.140889    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" event={"ID":"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2","Type":"ContainerStarted","Data":"a9901a1d1966d5940b8ad3edcae368cfc0ea3c3edb2724ab7d9e4d0c244bcfce"}
Sep 17 21:38:41.141116 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:41.140970    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" event={"ID":"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2","Type":"ContainerStarted","Data":"7cb3741ffe575a5b7fa0f61284ed7e0bacbdd2b76912b0e687f81a3999f27173"}
Sep 17 21:38:42.147951 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:42.147853    2571 generic.go:356] "Generic (PLEG): container finished" podID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerID="a9901a1d1966d5940b8ad3edcae368cfc0ea3c3edb2724ab7d9e4d0c244bcfce" exitCode=0
Sep 17 21:38:42.147951 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:42.147940    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" event={"ID":"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2","Type":"ContainerDied","Data":"a9901a1d1966d5940b8ad3edcae368cfc0ea3c3edb2724ab7d9e4d0c244bcfce"}
Sep 17 21:38:43.154981 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:43.154946    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" event={"ID":"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2","Type":"ContainerStarted","Data":"134d655fa0279eb6ef4c41aa868b0c662973f482738b6081e3b1c3be977924ea"}
Sep 17 21:38:43.155360 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:43.155002    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" event={"ID":"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2","Type":"ContainerStarted","Data":"4ce0710952b93885f397c966fafd50f974541c3a745d18fcf26be74daaaa3ea7"}
Sep 17 21:38:43.155360 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:43.155134    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:43.155360 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:43.155185    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:43.156469 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:43.156445    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 21:38:43.175525 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:43.175481    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" podStartSLOduration=3.123894036 podStartE2EDuration="4.17546232s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.051568284s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:38:43.174044836 +0000 UTC m=+2934.678522435" watchObservedRunningTime="2026-09-17 21:38:43.17546232 +0000 UTC m=+2934.679939915"
Sep 17 21:38:44.161799 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:44.161749    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 21:38:49.166621 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:49.166593    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:38:49.167074 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:49.167047    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 21:38:59.167083 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:38:59.167043    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 21:39:09.167956 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:39:09.167868    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 21:39:19.167542 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:39:19.167501    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 21:39:29.167051 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:39:29.167007    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 21:39:39.167411 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:39:39.167370    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 21:39:49.167188 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:39:49.167149    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 21:39:59.167057 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:39:59.167013    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 21:40:09.167783 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:09.167744    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:40:09.640746 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:09.640712    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"]
Sep 17 21:40:09.640962 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:09.640935    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kserve-container" containerID="cri-o://4ce0710952b93885f397c966fafd50f974541c3a745d18fcf26be74daaaa3ea7" gracePeriod=30
Sep 17 21:40:09.641055 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:09.640987    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kube-rbac-proxy" containerID="cri-o://134d655fa0279eb6ef4c41aa868b0c662973f482738b6081e3b1c3be977924ea" gracePeriod=30
Sep 17 21:40:10.593642 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:10.593602    2571 generic.go:356] "Generic (PLEG): container finished" podID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerID="134d655fa0279eb6ef4c41aa868b0c662973f482738b6081e3b1c3be977924ea" exitCode=2
Sep 17 21:40:10.594134 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:10.593679    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" event={"ID":"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2","Type":"ContainerDied","Data":"134d655fa0279eb6ef4c41aa868b0c662973f482738b6081e3b1c3be977924ea"}
Sep 17 21:40:14.162671 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:14.162629    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 17 21:40:14.800509 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:14.800483    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:40:14.919856 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:14.919777    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kube-rbac-proxy-sar-config\") pod \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") "
Sep 17 21:40:14.919856 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:14.919814    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-proxy-tls\") pod \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") "
Sep 17 21:40:14.919856 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:14.919852    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-cabundle-cert\") pod \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") "
Sep 17 21:40:14.920094 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:14.919896    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kserve-provision-location\") pod \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") "
Sep 17 21:40:14.920094 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:14.919973    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74qkb\" (UniqueName: \"kubernetes.io/projected/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kube-api-access-74qkb\") pod \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\" (UID: \"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2\") "
Sep 17 21:40:14.920241 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:14.920201    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" (UID: "d3e9829c-4fae-4c39-a7c9-72b3d7b612e2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:40:14.920311 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:14.920234    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" (UID: "d3e9829c-4fae-4c39-a7c9-72b3d7b612e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:40:14.920311 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:14.920265    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" (UID: "d3e9829c-4fae-4c39-a7c9-72b3d7b612e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:40:14.922264 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:14.922232    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kube-api-access-74qkb" (OuterVolumeSpecName: "kube-api-access-74qkb") pod "d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" (UID: "d3e9829c-4fae-4c39-a7c9-72b3d7b612e2"). InnerVolumeSpecName "kube-api-access-74qkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:40:14.922376 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:14.922300    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" (UID: "d3e9829c-4fae-4c39-a7c9-72b3d7b612e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:40:15.020959 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.020912    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-cabundle-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:40:15.020959 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.020952    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kserve-provision-location\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:40:15.020959 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.020962    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74qkb\" (UniqueName: \"kubernetes.io/projected/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kube-api-access-74qkb\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:40:15.021202 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.020973    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:40:15.021202 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.020982    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2-proxy-tls\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:40:15.621056 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.621015    2571 generic.go:356] "Generic (PLEG): container finished" podID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerID="4ce0710952b93885f397c966fafd50f974541c3a745d18fcf26be74daaaa3ea7" exitCode=0
Sep 17 21:40:15.621056 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.621059    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" event={"ID":"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2","Type":"ContainerDied","Data":"4ce0710952b93885f397c966fafd50f974541c3a745d18fcf26be74daaaa3ea7"}
Sep 17 21:40:15.621518 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.621070    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"
Sep 17 21:40:15.621518 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.621118    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff" event={"ID":"d3e9829c-4fae-4c39-a7c9-72b3d7b612e2","Type":"ContainerDied","Data":"7cb3741ffe575a5b7fa0f61284ed7e0bacbdd2b76912b0e687f81a3999f27173"}
Sep 17 21:40:15.621518 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.621146    2571 scope.go:117] "RemoveContainer" containerID="134d655fa0279eb6ef4c41aa868b0c662973f482738b6081e3b1c3be977924ea"
Sep 17 21:40:15.630510 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.630489    2571 scope.go:117] "RemoveContainer" containerID="4ce0710952b93885f397c966fafd50f974541c3a745d18fcf26be74daaaa3ea7"
Sep 17 21:40:15.638307 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.638288    2571 scope.go:117] "RemoveContainer" containerID="a9901a1d1966d5940b8ad3edcae368cfc0ea3c3edb2724ab7d9e4d0c244bcfce"
Sep 17 21:40:15.644378 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.644351    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"]
Sep 17 21:40:15.647043 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.647020    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-6vfff"]
Sep 17 21:40:15.647674 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.647657    2571 scope.go:117] "RemoveContainer" containerID="134d655fa0279eb6ef4c41aa868b0c662973f482738b6081e3b1c3be977924ea"
Sep 17 21:40:15.647975 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:40:15.647954    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134d655fa0279eb6ef4c41aa868b0c662973f482738b6081e3b1c3be977924ea\": container with ID starting with 134d655fa0279eb6ef4c41aa868b0c662973f482738b6081e3b1c3be977924ea not found: ID does not exist" containerID="134d655fa0279eb6ef4c41aa868b0c662973f482738b6081e3b1c3be977924ea"
Sep 17 21:40:15.648068 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.647987    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134d655fa0279eb6ef4c41aa868b0c662973f482738b6081e3b1c3be977924ea"} err="failed to get container status \"134d655fa0279eb6ef4c41aa868b0c662973f482738b6081e3b1c3be977924ea\": rpc error: code = NotFound desc = could not find container \"134d655fa0279eb6ef4c41aa868b0c662973f482738b6081e3b1c3be977924ea\": container with ID starting with 134d655fa0279eb6ef4c41aa868b0c662973f482738b6081e3b1c3be977924ea not found: ID does not exist"
Sep 17 21:40:15.648068 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.648011    2571 scope.go:117] "RemoveContainer" containerID="4ce0710952b93885f397c966fafd50f974541c3a745d18fcf26be74daaaa3ea7"
Sep 17 21:40:15.648292 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:40:15.648270    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce0710952b93885f397c966fafd50f974541c3a745d18fcf26be74daaaa3ea7\": container with ID starting with 4ce0710952b93885f397c966fafd50f974541c3a745d18fcf26be74daaaa3ea7 not found: ID does not exist" containerID="4ce0710952b93885f397c966fafd50f974541c3a745d18fcf26be74daaaa3ea7"
Sep 17 21:40:15.648336 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.648299    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce0710952b93885f397c966fafd50f974541c3a745d18fcf26be74daaaa3ea7"} err="failed to get container status \"4ce0710952b93885f397c966fafd50f974541c3a745d18fcf26be74daaaa3ea7\": rpc error: code = NotFound desc = could not find container \"4ce0710952b93885f397c966fafd50f974541c3a745d18fcf26be74daaaa3ea7\": container with ID starting with 4ce0710952b93885f397c966fafd50f974541c3a745d18fcf26be74daaaa3ea7 not found: ID does not exist"
Sep 17 21:40:15.648336 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.648322    2571 scope.go:117] "RemoveContainer" containerID="a9901a1d1966d5940b8ad3edcae368cfc0ea3c3edb2724ab7d9e4d0c244bcfce"
Sep 17 21:40:15.648595 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:40:15.648576    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9901a1d1966d5940b8ad3edcae368cfc0ea3c3edb2724ab7d9e4d0c244bcfce\": container with ID starting with a9901a1d1966d5940b8ad3edcae368cfc0ea3c3edb2724ab7d9e4d0c244bcfce not found: ID does not exist" containerID="a9901a1d1966d5940b8ad3edcae368cfc0ea3c3edb2724ab7d9e4d0c244bcfce"
Sep 17 21:40:15.648662 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:15.648601    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9901a1d1966d5940b8ad3edcae368cfc0ea3c3edb2724ab7d9e4d0c244bcfce"} err="failed to get container status \"a9901a1d1966d5940b8ad3edcae368cfc0ea3c3edb2724ab7d9e4d0c244bcfce\": rpc error: code = NotFound desc = could not find container \"a9901a1d1966d5940b8ad3edcae368cfc0ea3c3edb2724ab7d9e4d0c244bcfce\": container with ID starting with a9901a1d1966d5940b8ad3edcae368cfc0ea3c3edb2724ab7d9e4d0c244bcfce not found: ID does not exist"
Sep 17 21:40:16.670357 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:40:16.670315    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" path="/var/lib/kubelet/pods/d3e9829c-4fae-4c39-a7c9-72b3d7b612e2/volumes"
Sep 17 21:43:51.756641 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.756561    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"]
Sep 17 21:43:51.757108 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.756925    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="storage-initializer"
Sep 17 21:43:51.757108 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.756936    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="storage-initializer"
Sep 17 21:43:51.757108 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.756945    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kserve-container"
Sep 17 21:43:51.757108 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.756951    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kserve-container"
Sep 17 21:43:51.757108 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.756965    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kube-rbac-proxy"
Sep 17 21:43:51.757108 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.756971    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kube-rbac-proxy"
Sep 17 21:43:51.757108 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.757039    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kserve-container"
Sep 17 21:43:51.757108 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.757047    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3e9829c-4fae-4c39-a7c9-72b3d7b612e2" containerName="kube-rbac-proxy"
Sep 17 21:43:51.760368 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.760352    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:51.763409 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.763359    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-d588d2-dockercfg-n2qkh\""
Sep 17 21:43:51.763617 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.763602    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-d588d2-predictor-serving-cert\""
Sep 17 21:43:51.763868 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.763849    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:43:51.764496 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.764467    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:43:51.764496 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.764470    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-d588d2\""
Sep 17 21:43:51.764674 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.764502    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-d588d2-kube-rbac-proxy-sar-config\""
Sep 17 21:43:51.764826 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.764811    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:43:51.776413 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.776385    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"]
Sep 17 21:43:51.904997 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.904956    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99xfp\" (UniqueName: \"kubernetes.io/projected/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kube-api-access-99xfp\") pod \"isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") " pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:51.904997 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.904997    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-cabundle-cert\") pod \"isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") " pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:51.905280 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.905096    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-proxy-tls\") pod \"isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") " pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:51.905280 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.905133    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") " pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:51.905280 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:51.905261    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kserve-provision-location\") pod \"isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") " pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:52.006572 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:52.006510    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kserve-provision-location\") pod \"isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") " pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:52.006572 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:52.006568    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99xfp\" (UniqueName: \"kubernetes.io/projected/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kube-api-access-99xfp\") pod \"isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") " pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:52.006858 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:52.006585    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-cabundle-cert\") pod \"isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") " pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:52.006858 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:52.006621    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-proxy-tls\") pod \"isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") " pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:52.006858 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:52.006636    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") " pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:52.007033 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:52.007009    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kserve-provision-location\") pod \"isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") " pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:52.007313 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:52.007289    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-cabundle-cert\") pod \"isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") " pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:52.007430 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:52.007315    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") " pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:52.009227 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:52.009207    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-proxy-tls\") pod \"isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") " pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:52.015973 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:52.015946    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99xfp\" (UniqueName: \"kubernetes.io/projected/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kube-api-access-99xfp\") pod \"isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") " pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:52.070932 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:52.070871    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:43:52.347114 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:52.347087    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"]
Sep 17 21:43:52.349692 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:43:52.349666    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b3f4b3_fb30_4f7b_a424_d19fd3c006c9.slice/crio-5f9ecdf81cb8bf98341488a8b5c5327e3e09589b6b94a46d092c9a9636399721 WatchSource:0}: Error finding container 5f9ecdf81cb8bf98341488a8b5c5327e3e09589b6b94a46d092c9a9636399721: Status 404 returned error can't find the container with id 5f9ecdf81cb8bf98341488a8b5c5327e3e09589b6b94a46d092c9a9636399721
Sep 17 21:43:52.671583 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:52.671480    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt" event={"ID":"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9","Type":"ContainerStarted","Data":"76317fc2eddaa4be883ebd8d37a0b9fb1f6d831242edc8edba49cf42dd01fafd"}
Sep 17 21:43:52.671583 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:52.671538    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt" event={"ID":"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9","Type":"ContainerStarted","Data":"5f9ecdf81cb8bf98341488a8b5c5327e3e09589b6b94a46d092c9a9636399721"}
Sep 17 21:43:57.690336 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:57.690298    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt_f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9/storage-initializer/0.log"
Sep 17 21:43:57.690847 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:57.690365    2571 generic.go:356] "Generic (PLEG): container finished" podID="f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9" containerID="76317fc2eddaa4be883ebd8d37a0b9fb1f6d831242edc8edba49cf42dd01fafd" exitCode=1
Sep 17 21:43:57.690847 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:57.690456    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt" event={"ID":"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9","Type":"ContainerDied","Data":"76317fc2eddaa4be883ebd8d37a0b9fb1f6d831242edc8edba49cf42dd01fafd"}
Sep 17 21:43:58.697200 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:58.697175    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt_f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9/storage-initializer/0.log"
Sep 17 21:43:58.697690 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:43:58.697295    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt" event={"ID":"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9","Type":"ContainerStarted","Data":"971d7f07ca1327cf87e816618fcb182a9a1f34935efb0fcc0d3d0ad7ef1a6f4f"}
Sep 17 21:44:01.716398 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:01.716374    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt_f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9/storage-initializer/1.log"
Sep 17 21:44:01.716809 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:01.716794    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt_f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9/storage-initializer/0.log"
Sep 17 21:44:01.716859 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:01.716828    2571 generic.go:356] "Generic (PLEG): container finished" podID="f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9" containerID="971d7f07ca1327cf87e816618fcb182a9a1f34935efb0fcc0d3d0ad7ef1a6f4f" exitCode=1
Sep 17 21:44:01.716939 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:01.716916    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt" event={"ID":"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9","Type":"ContainerDied","Data":"971d7f07ca1327cf87e816618fcb182a9a1f34935efb0fcc0d3d0ad7ef1a6f4f"}
Sep 17 21:44:01.717017 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:01.717004    2571 scope.go:117] "RemoveContainer" containerID="76317fc2eddaa4be883ebd8d37a0b9fb1f6d831242edc8edba49cf42dd01fafd"
Sep 17 21:44:01.717421 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:01.717405    2571 scope.go:117] "RemoveContainer" containerID="76317fc2eddaa4be883ebd8d37a0b9fb1f6d831242edc8edba49cf42dd01fafd"
Sep 17 21:44:01.735478 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:44:01.735458    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt_kserve-ci-e2e-test(f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9)\"" pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt" podUID="f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9"
Sep 17 21:44:01.735711 ip-10-0-131-94 kubenswrapper[2571]: E0917 21:44:01.735671    2571 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt_kserve-ci-e2e-test_f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9_0 in pod sandbox 5f9ecdf81cb8bf98341488a8b5c5327e3e09589b6b94a46d092c9a9636399721 from index: no such id: '76317fc2eddaa4be883ebd8d37a0b9fb1f6d831242edc8edba49cf42dd01fafd'" containerID="76317fc2eddaa4be883ebd8d37a0b9fb1f6d831242edc8edba49cf42dd01fafd"
Sep 17 21:44:01.735814 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:01.735717    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76317fc2eddaa4be883ebd8d37a0b9fb1f6d831242edc8edba49cf42dd01fafd"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt_kserve-ci-e2e-test_f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9_0 in pod sandbox 5f9ecdf81cb8bf98341488a8b5c5327e3e09589b6b94a46d092c9a9636399721 from index: no such id: '76317fc2eddaa4be883ebd8d37a0b9fb1f6d831242edc8edba49cf42dd01fafd'"
Sep 17 21:44:02.723870 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:02.723838    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt_f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9/storage-initializer/1.log"
Sep 17 21:44:09.753559 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:09.753509    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"]
Sep 17 21:44:09.894708 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:09.894681    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt_f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9/storage-initializer/1.log"
Sep 17 21:44:09.898672 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:09.898654    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:44:09.949971 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:09.949931    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-cabundle-cert\") pod \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") "
Sep 17 21:44:09.949971 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:09.949977    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kserve-provision-location\") pod \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") "
Sep 17 21:44:09.950233 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:09.950023    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-proxy-tls\") pod \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") "
Sep 17 21:44:09.950233 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:09.950038    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kube-rbac-proxy-sar-config\") pod \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") "
Sep 17 21:44:09.950233 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:09.950069    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99xfp\" (UniqueName: \"kubernetes.io/projected/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kube-api-access-99xfp\") pod \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\" (UID: \"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9\") "
Sep 17 21:44:09.950407 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:09.950339    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9" (UID: "f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:44:09.950407 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:09.950369    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9" (UID: "f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:44:09.950510 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:09.950459    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9" (UID: "f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:44:09.952296 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:09.952271    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9" (UID: "f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:44:09.952396 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:09.952299    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kube-api-access-99xfp" (OuterVolumeSpecName: "kube-api-access-99xfp") pod "f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9" (UID: "f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9"). InnerVolumeSpecName "kube-api-access-99xfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:44:10.051495 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:10.051406    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-proxy-tls\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:44:10.051495 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:10.051435    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:44:10.051495 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:10.051444    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99xfp\" (UniqueName: \"kubernetes.io/projected/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kube-api-access-99xfp\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:44:10.051495 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:10.051455    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-cabundle-cert\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:44:10.051495 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:10.051463    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9-kserve-provision-location\") on node \"ip-10-0-131-94.ec2.internal\" DevicePath \"\""
Sep 17 21:44:10.774214 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:10.774184    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt_f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9/storage-initializer/1.log"
Sep 17 21:44:10.774630 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:10.774282    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"
Sep 17 21:44:10.774630 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:10.774290    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt" event={"ID":"f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9","Type":"ContainerDied","Data":"5f9ecdf81cb8bf98341488a8b5c5327e3e09589b6b94a46d092c9a9636399721"}
Sep 17 21:44:10.774630 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:10.774360    2571 scope.go:117] "RemoveContainer" containerID="971d7f07ca1327cf87e816618fcb182a9a1f34935efb0fcc0d3d0ad7ef1a6f4f"
Sep 17 21:44:10.812202 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:10.812173    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"]
Sep 17 21:44:10.816447 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:10.816422    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d588d2-predictor-7f96dcf4f4-hx4nt"]
Sep 17 21:44:12.669835 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:12.669804    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9" path="/var/lib/kubelet/pods/f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9/volumes"
Sep 17 21:44:56.398451 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:56.398420    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2h2gl_cfe78836-a877-45c6-9b50-c5a481497f71/global-pull-secret-syncer/0.log"
Sep 17 21:44:56.562080 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:56.562041    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nm7bs_9ada893c-8b3a-46bf-a871-060c70e9cf88/konnectivity-agent/0.log"
Sep 17 21:44:56.642837 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:44:56.642808    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-94.ec2.internal_26576e07a04b23124eb74ce2ed7324c3/haproxy/0.log"
Sep 17 21:45:00.287615 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.287586    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_009566dd-7ba3-4549-8b14-8c62863f663d/alertmanager/0.log"
Sep 17 21:45:00.316847 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.316820    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_009566dd-7ba3-4549-8b14-8c62863f663d/config-reloader/0.log"
Sep 17 21:45:00.356673 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.356648    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_009566dd-7ba3-4549-8b14-8c62863f663d/kube-rbac-proxy-web/0.log"
Sep 17 21:45:00.394640 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.394603    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_009566dd-7ba3-4549-8b14-8c62863f663d/kube-rbac-proxy/0.log"
Sep 17 21:45:00.428643 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.428616    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_009566dd-7ba3-4549-8b14-8c62863f663d/kube-rbac-proxy-metric/0.log"
Sep 17 21:45:00.467444 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.467415    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_009566dd-7ba3-4549-8b14-8c62863f663d/prom-label-proxy/0.log"
Sep 17 21:45:00.509167 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.509115    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_009566dd-7ba3-4549-8b14-8c62863f663d/init-config-reloader/0.log"
Sep 17 21:45:00.553374 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.553293    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-rgxgj_5696194e-5b60-4427-9147-1c42eb17a7ed/cluster-monitoring-operator/0.log"
Sep 17 21:45:00.579535 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.579506    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-x4dbc_ef08f252-a2f1-45ee-94a9-bb148e841f90/kube-state-metrics/0.log"
Sep 17 21:45:00.603631 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.603603    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-x4dbc_ef08f252-a2f1-45ee-94a9-bb148e841f90/kube-rbac-proxy-main/0.log"
Sep 17 21:45:00.630347 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.630324    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-x4dbc_ef08f252-a2f1-45ee-94a9-bb148e841f90/kube-rbac-proxy-self/0.log"
Sep 17 21:45:00.668522 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.668496    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-984fd59-brmlv_d5ed028c-8213-427d-a77f-f4d78e7fd1da/metrics-server/0.log"
Sep 17 21:45:00.700379 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.700351    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-dt6xb_34853159-04ae-4640-b9cf-804bcc9ab35f/monitoring-plugin/0.log"
Sep 17 21:45:00.817672 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.817644    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dbcl5_872e9212-3c43-4ef7-94aa-b27904af77c0/node-exporter/0.log"
Sep 17 21:45:00.843047 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.843024    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dbcl5_872e9212-3c43-4ef7-94aa-b27904af77c0/kube-rbac-proxy/0.log"
Sep 17 21:45:00.867144 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.867121    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dbcl5_872e9212-3c43-4ef7-94aa-b27904af77c0/init-textfile/0.log"
Sep 17 21:45:00.982061 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:00.982028    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-ksvcr_41e43ece-8336-4b4c-9355-feaa49cd2d2c/kube-rbac-proxy-main/0.log"
Sep 17 21:45:01.009012 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:01.008987    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-ksvcr_41e43ece-8336-4b4c-9355-feaa49cd2d2c/kube-rbac-proxy-self/0.log"
Sep 17 21:45:01.031893 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:01.031871    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-ksvcr_41e43ece-8336-4b4c-9355-feaa49cd2d2c/openshift-state-metrics/0.log"
Sep 17 21:45:01.302910 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:01.302841    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-2chxs_51c1c187-47e0-48a8-aeb8-1f46821144ac/prometheus-operator-admission-webhook/0.log"
Sep 17 21:45:02.828904 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:02.828873    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-xlfkt_bdc1fd31-eb35-4d76-9364-c26bacdcf59f/networking-console-plugin/0.log"
Sep 17 21:45:03.780066 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:03.780037    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-677fdbbb87-4h64j_24ff5f9f-91c5-4927-910c-974c795bd13b/console/0.log"
Sep 17 21:45:04.447702 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.447667    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"]
Sep 17 21:45:04.448085 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.448024    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9" containerName="storage-initializer"
Sep 17 21:45:04.448085 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.448035    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9" containerName="storage-initializer"
Sep 17 21:45:04.448156 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.448107    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9" containerName="storage-initializer"
Sep 17 21:45:04.448156 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.448118    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b3f4b3-fb30-4f7b-a424-d19fd3c006c9" containerName="storage-initializer"
Sep 17 21:45:04.452339 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.452320    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:04.455989 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.455963    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4cltr\"/\"default-dockercfg-9rqsm\""
Sep 17 21:45:04.456124 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.456010    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4cltr\"/\"kube-root-ca.crt\""
Sep 17 21:45:04.456124 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.456013    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4cltr\"/\"openshift-service-ca.crt\""
Sep 17 21:45:04.461245 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.461219    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"]
Sep 17 21:45:04.518336 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.518291    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cea2ea0-62a6-432a-b74a-5e645b33876c-sys\") pod \"perf-node-gather-daemonset-scz8m\" (UID: \"0cea2ea0-62a6-432a-b74a-5e645b33876c\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:04.518336 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.518339    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ssw\" (UniqueName: \"kubernetes.io/projected/0cea2ea0-62a6-432a-b74a-5e645b33876c-kube-api-access-b4ssw\") pod \"perf-node-gather-daemonset-scz8m\" (UID: \"0cea2ea0-62a6-432a-b74a-5e645b33876c\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:04.518597 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.518370    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0cea2ea0-62a6-432a-b74a-5e645b33876c-podres\") pod \"perf-node-gather-daemonset-scz8m\" (UID: \"0cea2ea0-62a6-432a-b74a-5e645b33876c\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:04.518597 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.518444    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cea2ea0-62a6-432a-b74a-5e645b33876c-lib-modules\") pod \"perf-node-gather-daemonset-scz8m\" (UID: \"0cea2ea0-62a6-432a-b74a-5e645b33876c\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:04.518597 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.518477    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0cea2ea0-62a6-432a-b74a-5e645b33876c-proc\") pod \"perf-node-gather-daemonset-scz8m\" (UID: \"0cea2ea0-62a6-432a-b74a-5e645b33876c\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:04.619661 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.619621    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0cea2ea0-62a6-432a-b74a-5e645b33876c-podres\") pod \"perf-node-gather-daemonset-scz8m\" (UID: \"0cea2ea0-62a6-432a-b74a-5e645b33876c\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:04.619823 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.619675    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cea2ea0-62a6-432a-b74a-5e645b33876c-lib-modules\") pod \"perf-node-gather-daemonset-scz8m\" (UID: \"0cea2ea0-62a6-432a-b74a-5e645b33876c\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:04.619823 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.619703    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0cea2ea0-62a6-432a-b74a-5e645b33876c-proc\") pod \"perf-node-gather-daemonset-scz8m\" (UID: \"0cea2ea0-62a6-432a-b74a-5e645b33876c\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:04.619823 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.619740    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cea2ea0-62a6-432a-b74a-5e645b33876c-sys\") pod \"perf-node-gather-daemonset-scz8m\" (UID: \"0cea2ea0-62a6-432a-b74a-5e645b33876c\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:04.619823 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.619756    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ssw\" (UniqueName: \"kubernetes.io/projected/0cea2ea0-62a6-432a-b74a-5e645b33876c-kube-api-access-b4ssw\") pod \"perf-node-gather-daemonset-scz8m\" (UID: \"0cea2ea0-62a6-432a-b74a-5e645b33876c\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:04.619823 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.619806    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0cea2ea0-62a6-432a-b74a-5e645b33876c-podres\") pod \"perf-node-gather-daemonset-scz8m\" (UID: \"0cea2ea0-62a6-432a-b74a-5e645b33876c\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:04.620017 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.619840    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cea2ea0-62a6-432a-b74a-5e645b33876c-lib-modules\") pod \"perf-node-gather-daemonset-scz8m\" (UID: \"0cea2ea0-62a6-432a-b74a-5e645b33876c\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:04.620017 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.619845    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cea2ea0-62a6-432a-b74a-5e645b33876c-sys\") pod \"perf-node-gather-daemonset-scz8m\" (UID: \"0cea2ea0-62a6-432a-b74a-5e645b33876c\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:04.620017 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.619845    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0cea2ea0-62a6-432a-b74a-5e645b33876c-proc\") pod \"perf-node-gather-daemonset-scz8m\" (UID: \"0cea2ea0-62a6-432a-b74a-5e645b33876c\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:04.628806 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.628783    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ssw\" (UniqueName: \"kubernetes.io/projected/0cea2ea0-62a6-432a-b74a-5e645b33876c-kube-api-access-b4ssw\") pod \"perf-node-gather-daemonset-scz8m\" (UID: \"0cea2ea0-62a6-432a-b74a-5e645b33876c\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:04.763059 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:04.762959    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:05.033860 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:05.033714    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"]
Sep 17 21:45:05.036181 ip-10-0-131-94 kubenswrapper[2571]: W0917 21:45:05.036147    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0cea2ea0_62a6_432a_b74a_5e645b33876c.slice/crio-86cbd78bd5a8bb49936ad3a50869f0e0c18da377be2bdcb33f176eb466191e1e WatchSource:0}: Error finding container 86cbd78bd5a8bb49936ad3a50869f0e0c18da377be2bdcb33f176eb466191e1e: Status 404 returned error can't find the container with id 86cbd78bd5a8bb49936ad3a50869f0e0c18da377be2bdcb33f176eb466191e1e
Sep 17 21:45:05.352295 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:05.352270    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mp5gn_325ad1ea-7622-490d-82c3-ff73e8ff5dda/dns/0.log"
Sep 17 21:45:05.373786 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:05.373763    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mp5gn_325ad1ea-7622-490d-82c3-ff73e8ff5dda/kube-rbac-proxy/0.log"
Sep 17 21:45:05.465780 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:05.465752    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n779j_0cc2bef5-b9e2-4a70-9905-f425604d5f3c/dns-node-resolver/0.log"
Sep 17 21:45:06.045801 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:06.045767    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m" event={"ID":"0cea2ea0-62a6-432a-b74a-5e645b33876c","Type":"ContainerStarted","Data":"5a62d0ab0e68e8e9e497eea1efdd9f74c2fafe25ed05ac5d81ee5ea5bd6f3615"}
Sep 17 21:45:06.045995 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:06.045820    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m" event={"ID":"0cea2ea0-62a6-432a-b74a-5e645b33876c","Type":"ContainerStarted","Data":"86cbd78bd5a8bb49936ad3a50869f0e0c18da377be2bdcb33f176eb466191e1e"}
Sep 17 21:45:06.045995 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:06.045841    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:06.050489 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:06.050461    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qmv8f_d934ceba-229e-4206-bb5f-31b8e8af019f/node-ca/0.log"
Sep 17 21:45:06.063560 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:06.063492    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m" podStartSLOduration=2.063472315 podStartE2EDuration="2.063472315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:45:06.061413253 +0000 UTC m=+3317.565890850" watchObservedRunningTime="2026-09-17 21:45:06.063472315 +0000 UTC m=+3317.567949913"
Sep 17 21:45:07.157292 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:07.157264    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f2bdn_07aff763-5b60-4dda-89de-f72e2896a715/serve-healthcheck-canary/0.log"
Sep 17 21:45:07.512394 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:07.512314    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-4rvqw_8bf9e322-db1c-4373-986b-d4d95d2dad3c/insights-operator/0.log"
Sep 17 21:45:07.513205 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:07.513185    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-4rvqw_8bf9e322-db1c-4373-986b-d4d95d2dad3c/insights-operator/1.log"
Sep 17 21:45:07.533312 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:07.533286    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5hc26_92cc791b-7e51-4256-9e52-0dbf7d51ee0f/kube-rbac-proxy/0.log"
Sep 17 21:45:07.554858 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:07.554834    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5hc26_92cc791b-7e51-4256-9e52-0dbf7d51ee0f/exporter/0.log"
Sep 17 21:45:07.576595 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:07.576542    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5hc26_92cc791b-7e51-4256-9e52-0dbf7d51ee0f/extractor/0.log"
Sep 17 21:45:10.011955 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:10.011922    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-66f4b8957f-csj4z_c297151d-b138-4f59-910c-72f69ae716e6/manager/0.log"
Sep 17 21:45:10.063290 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:10.063258    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-22vrx_382c2a30-c5dd-42a1-b5ef-d990faa733fa/server/0.log"
Sep 17 21:45:10.530692 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:10.530661    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-zsdnx_d53ab2e6-bf64-43ae-afe7-31bb00121e4a/seaweedfs/0.log"
Sep 17 21:45:10.553169 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:10.553113    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-k9mr6_9f833c5f-aebb-478d-ab54-abab238ee436/seaweedfs-tls-custom/0.log"
Sep 17 21:45:10.574923 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:10.574897    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-zvd7g_ada3840e-1c1c-47e8-81c6-d73f67164067/seaweedfs-tls-serving/0.log"
Sep 17 21:45:12.061876 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:12.061844    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-scz8m"
Sep 17 21:45:15.313172 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:15.313140    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-xbdx5_374b4cca-e914-4b51-a833-32dcbcd655f2/kube-storage-version-migrator-operator/1.log"
Sep 17 21:45:15.313890 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:15.313867    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-xbdx5_374b4cca-e914-4b51-a833-32dcbcd655f2/kube-storage-version-migrator-operator/0.log"
Sep 17 21:45:16.350214 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:16.350184    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9dxv5_bf6c96de-21ca-45f6-b2f0-8b9b3b65411b/kube-multus/0.log"
Sep 17 21:45:16.372016 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:16.371987    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47dgq_16b58fb3-0631-4fc1-894a-1d1f7a48b468/kube-multus-additional-cni-plugins/0.log"
Sep 17 21:45:16.394281 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:16.394255    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47dgq_16b58fb3-0631-4fc1-894a-1d1f7a48b468/egress-router-binary-copy/0.log"
Sep 17 21:45:16.415146 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:16.415119    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47dgq_16b58fb3-0631-4fc1-894a-1d1f7a48b468/cni-plugins/0.log"
Sep 17 21:45:16.435131 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:16.435103    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47dgq_16b58fb3-0631-4fc1-894a-1d1f7a48b468/bond-cni-plugin/0.log"
Sep 17 21:45:16.458314 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:16.458284    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47dgq_16b58fb3-0631-4fc1-894a-1d1f7a48b468/routeoverride-cni/0.log"
Sep 17 21:45:16.478789 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:16.478768    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47dgq_16b58fb3-0631-4fc1-894a-1d1f7a48b468/whereabouts-cni-bincopy/0.log"
Sep 17 21:45:16.500236 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:16.500204    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-47dgq_16b58fb3-0631-4fc1-894a-1d1f7a48b468/whereabouts-cni/0.log"
Sep 17 21:45:16.984043 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:16.984017    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6742m_d2215eda-bca2-49dd-9419-90f9736fdf41/network-metrics-daemon/0.log"
Sep 17 21:45:17.002936 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:17.002915    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6742m_d2215eda-bca2-49dd-9419-90f9736fdf41/kube-rbac-proxy/0.log"
Sep 17 21:45:18.093522 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:18.093496    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8wlt5_89ae281b-8074-4ef8-a264-991b56d6e3ff/ovn-controller/0.log"
Sep 17 21:45:18.111072 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:18.111051    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8wlt5_89ae281b-8074-4ef8-a264-991b56d6e3ff/ovn-acl-logging/0.log"
Sep 17 21:45:18.135313 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:18.135278    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8wlt5_89ae281b-8074-4ef8-a264-991b56d6e3ff/kube-rbac-proxy-node/0.log"
Sep 17 21:45:18.156893 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:18.156864    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8wlt5_89ae281b-8074-4ef8-a264-991b56d6e3ff/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 21:45:18.176595 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:18.176568    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8wlt5_89ae281b-8074-4ef8-a264-991b56d6e3ff/northd/0.log"
Sep 17 21:45:18.200393 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:18.200370    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8wlt5_89ae281b-8074-4ef8-a264-991b56d6e3ff/nbdb/0.log"
Sep 17 21:45:18.224864 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:18.224841    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8wlt5_89ae281b-8074-4ef8-a264-991b56d6e3ff/sbdb/0.log"
Sep 17 21:45:18.270446 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:18.270415    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8wlt5_89ae281b-8074-4ef8-a264-991b56d6e3ff/ovnkube-controller/0.log"
Sep 17 21:45:19.521906 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:19.521875    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-8g7bq_fb1ed212-5c4f-4b71-b7f7-6057ba0eb12b/check-endpoints/0.log"
Sep 17 21:45:19.544777 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:19.544746    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-csv4n_a47b9a17-f95a-4e22-9584-a457ebbefdaf/network-check-target-container/0.log"
Sep 17 21:45:20.595604 ip-10-0-131-94 kubenswrapper[2571]: I0917 21:45:20.595576    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-mw75s_fa88803e-75a6-46c7-b9b4-82e86bbf3540/iptables-alerter/0.log"